The core of AI defect detection systems in production is automated visual inspection based on deep learning (DL) models, which recognize surface defects, deformations, etc. This article is based on the practical experience and research of MobiDev’s AI team, and explains how to implement AI visual inspection for defect detection in manufacturing lines.
Deep Learning at the Core of Automated Defect Detection
Advanced AI defect detection for complex productions should seamlessly identify various types of defects, distinguish between critical defects of the product that make it unusable and non-critical defects that are allowed by the manufacturer’s standards, etc. Therefore, the automated visual inspection must take into account many factors, including the conclusions of previous checks. At the same time, such an approach requires an enormous amount of data. The greater the range of products and their possible defects, the more samples are needed.
In essence, the system requires an imitation of the work of an experienced, qualified human specialist. This can be achieved by using a deep learning approach to teach machines to learn by example while dealing with an immense amount of data.
Then, starting from the basic logic created during initial training, AI engineers can continuously improve deep neural networks using new data, including images, for this purpose.
By providing a neural network with labeled examples of specific types of data, it’s possible to extract common patterns between those examples. This helps to classify future pieces of information. As a result, an AI inspection system with the integration of deep learning algorithms may distinguish parts, anomalies, and characters, just as a person does during visual inspection in manufacturing.
That is why deep learning-based visual inspection systems are today the optimal solution for detecting complex defects. They not only address complex surfaces and cosmetic flaws but also generalize and conceptualize the parts’ surfaces. Thus, the AI-based surface defect inspection works reliably and is sufficiently accurate in the conditions for which it was designed.
A crucial factor is the relevance of the data used for the AI-powered visual inspection system. Only labeled images are suitable, which are taken in the same conditions that are actually present on the production line, where the automated defect detection will be performed (same distance, lighting, angle, image quality, etc.). The dataset must be not only large enough but also balanced, with a fair number of defective samples.
Data Quantity and Quality Impact Accuracy of AI Defect Detection Systems
If you’re tasked to build a visual inspection software for a manufacturing line, you should first select a deep learning model architecture that fits your issue. Next, you train such a model with examples of defects that it must detect and with examples of normal products without defects.
With enough data, the neural network will eventually detect defects without any additional instructions.
We stress the need for sufficient data for a reason. Data sufficiency and relevance are key conditions for creating AI defect detection software.
Gathering datasets is the homework that, in most cases, the manufacturer has to do. Gathered data must correspond to the tasks you set for the AI inspection system and cover the parameters of your production. Ideally, data should be gathered on the same production line in the same conditions under which the defect detection software will operate.
After the initial deployment of the automated visual inspection system, you can gather more data while the production line is running. With new data, it is possible to train a better version of the model to improve the accuracy of AI defect detection. This will require labeling the new data and evaluating the new model to compare it to the old one. But even the initial dataset must be complete enough to train a deep learning model.
Let’s consider a case of detecting defects of spare parts or fasteners. For that, it is necessary to select examples of defects for the entire range of products. For each type, brand, and size of items, you need to collect examples of defects. If the parts are made of different materials, then, accordingly, it is necessary to expand the data sets. At the same time, the process can be focused not only on surface defects and mechanical damage but also on problems with coloring, for example.
Similar preparation is carried out before creating a computer vision defect detection system on transport networks, for example, railway tracks or pipelines. Large arrays of data are needed, in videos and images formats, which set samples of good (defect-free) and bad (defective) examples.
Incidentally, we should note that such image processing defect detection is even more complicated. After all, to determine the exact location of the defect, it requires linking to navigation systems and additional calculations based on measurements (e.g., degree of roll, amount of superelevation/cant, etc.).
Hence, the accuracy of an AI inspection system is derived from the quantity and quality of the data gathered for its development. So, the key to the success of projects of this type is in the hands of the manufacturer.
Building an AI-driven Defect Detection System: Step by Step
Let’s navigate now through the six stages of development and implementation of an AI visual inspection system for defect detection.
1. BUSINESS ANALYSIS STAGE
The development of AI-based visual control software, like any AI app development process, should start with a business analysis. It is necessary to comprehensively and clearly outline the client’s specific business problem, which will be solved by the visual inspection software product based on the available data.
When setting development goals and gathering product requirements, the core need is to determine what kind of defects the system should detect. It is also essential to find the answers to the following questions:
- Does data for deep learning model development exist, including images of “good” and “bad” products and the different types of defects? This question can be considered key.
- What is the automatic visual inspection system environment?
- Should the inspection be real-time or deferred?
- How thoroughly should the visual inspection system detect defects, and should it distinguish them by type?
- Is there any existing software that integrates the visual inspection feature, or does it require development from scratch?
- How should the system notify the user(s) about detected defects?
- Should the visual inspection system record defect detection statistics?
Having formed a set of documents that define the aims and tasks of the project, you can move on.
After conducting the business analysis stage and answering the questions we outlined above, you will be able to assess the business value that can be realistically obtained by implementing AI-based defect detection.
2. DEEP LEARNING METHOD SELECTION STAGE
This stage involves a fork in the road. It is necessary to make a decisive choice for the project between the search for a pretrained model, if it is possible to choose one, and the building of a custom model from scratch. The selection of a deep learning model development approach depends on the complexity of your task, required delivery time, and budget limitations. There are three possible approaches:
2.1. Using pretrained models
A pretrained model is an already-created deep learning model that accomplishes tasks similar to the one you want to perform. In this case, you do not have to train your model from scratch.
Using a pretrained model provides significant time and cost savings. However, such a model may not 100% match your tasks, and even in this situation, it will not be possible to do without fine tuning.
AI engineers can use Transfer Learning to take a pretrained model previously trained on a large dataset and fine tune it for your specific issue based on the features of your production line. This approach works well to improve metrics if such a model is trained on at least somewhat similar data.
Meanwhile, there is a pitfall on this path. For very specific industrial use cases, it can be impossible to find a pretrained model. As noted above, the use of non-specific data will not bring any benefit. Nevertheless, if there are many companies in your field of production with the same technology as yours, the chances of finding and implementing a pretrained model are greater.
2.2. Deep Learning model development from scratch
This method is ideal for complex visual inspection systems for advanced manufacturing use cases. Custom development, including model training, can be time and effort-intensive, but the results are worth it.
The custom model is, of course, more suitable for solving specific business issues. After all, such a product can be endowed with all the necessary functionality and optimized for the manufacturer’s needs.
However, we have to admit that this way of obtaining an AI inspection system is challenging and the main challenge is data availability.
Data in small volumes will not bring the proper effect in detecting defects since it will not be enough for gradual training and improvement of deep learning systems. The inability to obtain the necessary data can be an insurmountable obstacle for newly launched businesses. This is why AI defect detection development is more affordable to production pipelines that run long enough to collect data.
In contrast, companies that have just appeared or are just planning to implement new production lines do not yet have the necessary data and cannot always acquire them.
Thus, data sourcing is the cornerstone for building an AI-based visual inspection system from scratch. An existing manufacturing business may have its own datasets for machine learning. AI experts, in turn, may help identify ways to supplement and enrich these datasets. A newly started manufacturing company may consider purchasing or ordering data.
The availability of relevant data significantly increases the probability of building a working system, both in custom development and based on an off-the-shelf AI visual inspection solution. However, the more unique your product`s defects are, the less likely they are to be covered by pretrained models. This should be considered when choosing between custom development and ready-made software. The two examples below will show how it works in practice.
- Upon visual inspection for pharmaceutical manufacturing, it is necessary to differentiate air bubbles from particles in products like highly viscous parental solutions. The presence of bubbles is the only defect category here. So, the required dataset will not be extensive. The optimal approach to building computer vision defect detection software might be to use a pretrained model over training from scratch.
- Or, let’s say that we’re developing a visual inspection model for defect detection in construction. The main focus is to identify defects on the walls. Defect categories might be incredibly diverse, from peeling paint and mold to wall cracks. Therefore, an extensive dataset is necessary to obtain accurate visual inspection results. A pre-trained model approach may be viable in some cases. But still, the optimal approach here would be to develop an instance segmentation-based model from scratch.
3. DATA GATHERING AND PREPARATION STAGE
From our extensive experience developing AI apps in various domains, we know that the probability of getting perfect ready-made datasets is the same as finding a faceted diamond in a diamond mine.
Therefore, our AI engineers gather and prepare the data required to train a future model before deep learning model development starts.
There are several possible sources of obtaining such data, but the most common are:
- an existing video record provided by a manufacturer
- open-source video records applicable for defined purposes
- gathering data from scratch according to deep learning model requirements
Much of the data for the AI visual inspection models comes from video footage captured on production lines. The video record’s quality does matter. Higher-quality data will lead to more accurate results.
Preparation of gathered data includes at least two essential components:
- Data Labeling
- Exploratory Data Analysis (EDA).
It is necessary to label the data. Specialists should label each piece of collected data with informative tags. In this way, images are identified, and their details are characterized. This is all necessary for training the model being created. To develop a visual inspection system, for example, such tags may contain an indication of the presence of a defect and its type, and perhaps some helpful comments, etc.
Labeling of visual data includes, in particular, the following:
- Data labeling for classification consists only in putting images into some categories
- Detection needs labeling bounding boxes with the objects of classes we need
- Segmentation requires selecting very specific areas of the image
Having gathered the data, AI engineers prepare it for modeling, improving the quality. They clean it, check it for anomalies, and ensure its relevance to form working datasets for developing and training a deep learning model.
In Exploratory Data Analysis (EDA), AI engineers perform statistical analysis of images to find the proper way to normalize them and check whether the dataset is balanced. In data cleaning, specialists check data for errors, lacking values, and inconsistencies, and identify and remove outliers.
At this stage of the data work, it is critical to identify and eliminate bias in the data, i.e., the overweighting or overrepresentation of certain elements of the dataset. The use of a balanced, unbiased dataset increases the accuracy of the deep learning model’s defect detection. Data Quality Assessment at the end of EDA allows you to make sure the data set is suitable for further steps in building the AI visual inspection system.
4. DEEP LEARNING MODEL DEVELOPMENT STAGE
When developing custom visual inspection models, AI engineers use one or, if necessary, several computer vision algorithms, which depends on business goals and specific issues.
To develop AI defect detection software, depending on business requirements, it is necessary to focus on different types of computer vision problems:
- Classification: It works well for the case when we have only one product in the frame, and we need only to know if it is good or defective
- Detection: It takes place when we need bounding boxes for each product
- Segmentation: We deal with it when we need to understand which pixels belong to which object
Creating one automated visual inspection system may require coping with several types of computer vision problems. We can, for example, combine a detection model to find separate products and then use a classification model to understand if it has any defects.
Many factors influence the choice of a deep learning algorithm(s). These include:
- Business goals
- Size of objects/defects
- Lighting conditions
- Number of products to inspect
- Types of defects
- Resolution of images
Depending on the agreed parameters of the model, AI engineers design the architecture of a deep neural network. The created neural network architecture and prepared large datasets of labeled data are necessary to train a deep learning model for AI visual inspection.
5. TRAINING & EVALUATION STAGE
The next step is to train the built AI visual inspection model. For these purposes, AI professionals split the data set into training, validation, and testing datasets.
For example, in our case, a test dataset can be built from video recordings made on the production line. This requires getting separate frames from videos and labeling them. Such a set of labeled images is suitable for use in the creation of an automated visual inspection system.
Fundamental to model training is the loss function, which compares the target and predicted output values, thereby showing how well the neural network models the training data.
Completing the training and evaluation stage, AI engineers validate and evaluate the performance and result accuracy of the model.
6. DEPLOYMENT & IMPROVEMENT STAGE
When deploying a vision inspection system, the software and hardware architectures must match the model’s capabilities.
Let us briefly describe the main elements of the AI visual inspection system.
Software
Visual inspection software includes web modules for data transmission and a module for neural network processing. The key parameter here is data storage. There are three common ways to store data: on a local server, a cloud streaming service, or on serverless architecture.
The choice of data storage solution often depends on the functionality of the deep learning model. For example, if a visual inspection system uses a large data set, a cloud streaming service may be the optimal choice.
Hardware
Depending on the industry and automation processes, devices required to integrate a visual inspection system may include:
- Camera: The main camera option is real-time video streaming. Some examples include IP and CCTV.
- Gateway: Both dedicated hardware appliances and software programs must work well for a visual inspection system.
- CPU / GPU: If real-time results are necessary, a GPU would be a better choice than a CPU. The point is that the GPU boasts a faster processing speed when it comes to image-based deep learning models. It is possible to optimize a CPU for operating the visual inspection model, but not for training. An example of an optimal GPU might be the Jetson Nano.
- Photometer (optional): Depending on the lighting conditions, the visual inspection system may require photometers.
- Colorimeter (optional): When detecting color and luminance in light sources, imaging colorimeters have consistently high spatial resolutions, allowing for detailed visual inspections.
- Thermographic camera (optional): In the case of automated inspection of steam/water pipelines and facilities, it is a good idea to have thermographic camera data. Thermographic camera data provides valuable information for heat/steam/water leakage detection. Thermal camera data is also useful for heat insulation inspection.
- Drones (optional): Nowadays, it is hard to imagine automated inspection of hard-to-reach areas without drones. Some examples are building internals, gas pipelines, tanker visual inspection, and rocket/shuttle inspection. Drones may be equipped with high-resolution cameras that can do real-time defect detection.
The complete set of hardware for AI visual inspection for defect detection has many options. For example, there are many manufacturer requests to use for visual inspection purposes portable devices like Raspberry PI or Arduino computers. Likewise, in some situations of practical application of image processing defect detection, it is useful that compact but powerful Nvidia Jetson Nano or Nvidia Jetson Xavier can work in pairs to perform optical analysis.
A valuable feature of deep learning models is that they can be refined after deployment. A deep learning approach can increase the accuracy of the neural network through the iterative gathering of new data and model re-training. The result is a perfected visual inspection model that can be trained by increasing the amount of data during operation.
Key Takeaways: Highlights of AI-Powered Visual Inspection
You can get acquainted with the visual summary of the AI defect detection system building, which will help you understand the main turning points of the development project.
Start Your Development Journey with MobiDev
AI defect detection system development requires thorough preparation but can advance your company. Such actions will help to reduce losses from defective products, improve the customer experience, and strengthen the company’s market position.
Contact MobiDev to discuss possible approaches to developing an AI inspection system that fits your company needs. Our experts will guide you through all stages of the visual inspection software development cycle, from business analysis and design to deployment and support.
