March 14, 2025
This article demonstrates the development and implementation of a solution for Label AI, an automated compliance verification system for label/films validation for the beverage manufacturing client. The solution integrates innovative AI technologies, including Azure Document Intelligence, Computer Vision, and OpenCV, to optimize label/film verification and ensure regulatory compliance. This article gives in-depth analysis of the problem, development phases, challenges & resolutions, and key takeaways from the solution implementation.
Our client, A beverage manufacturer was facing considerable challenges with manual label/film verification, which was prone to human errors due to manual processes, operational delays, and compliance risks. The ineffectiveness in the manual approach led to frequent flaws, increased costs, and regulatory non-compliance issues.
The client, encountered significant difficulties with the existing label verification process which was handled manually. This current manual approach was not only laborious but also highly vulnerable to human error, leading to frequent flaws that posed significant compliance risks and operational delays. The need for a more efficient and reliable solution became vital to maintain market competitiveness and regulatory compliance.
key Objectives
The objective of Label AI was to develop an automated, scalable, and reliable system that:
- Enhances label verification accuracy using AI-powered OCR.
- Reduce operational delays and manual dependency.
- Make sure compliance with industry regulations.
- Integrates flawlessly with existing enterprise systems for real-time status and update.
The Planning / Implementation Approach
To address the existing challenges, we started with a detailed analysis phase where we thoroughly examined the client’s current manual processes. This involved discussions with the client to capture all existing details and challenges. Our objective was to understand the scope of the flaws and to determine the specific areas where automation could introduce substantial improvements. Based on this detailed analysis, we came up with a strategic plan that leveraged innovative technology solutions aimed at automating the label/film verification process to enhance accuracy, compliance, and performance. The development was divided into several key phases:
Technology Selection and Configuration
After analysis and research, a suite of Azure technologies was identified to meet the specific needs of the project:
- Azure Document Intelligence: for advanced OCR capabilities, customized to accurately extract text from complex label backgrounds.
- Azure Computer Vision: for its robust image analysis tools that automate the verification of text localization, label integrity, and compliance with graphical layout regulations.
- Azure App Service and Azure Container Registry: were chosen to host the application and manage Docker container images respectively, ensuring seamless deployment and scalability.
- OpenCV: for tasks requiring higher precision in image processing, such as edge detection and color analysis.
- A custom spell checker: developed to ensure text accuracy, utilizing natural language processing to correct OCR misreads and validate spelling against an industry-specific dictionary.
System Integration and Testing
- Custom OCR models were meticulously crafted and iteratively tested to handle the diverse typography found on beverage labels.
- Integration of Azure Computer Vision with OpenCV allowed for the establishment of a robust image processing pipeline capable of handling complex tasks like contour detection.
- Continuous Integration and Continuous Deployment (CI/CD) pipelines were set up using Azure DevOps.
Operational Deployment
- The automated system was fully integrated with the client’s existing ERP and inventory management systems via secure API gateways, enabling real-time updates and data synchronization.
- A comprehensive dashboard using Power BI was deployed to provide stakeholders with real-time insights into the system’s performance, including validation accuracy.
Implementation Phases
The implementation phase was systematically divided into several key steps:
After thoroughly analyzing the requirement, Royal Cyber suggested a combination of azure document intelligence along with computer vision and AI-based text validation to meet the client needs.
Our data experts chose Azure document intelligence because of the quality of its OCR, since the whole pipeline is dependent on the text that is extracted and the segmentation of the text that is performed by the custom OCR model. If these extractions by the model are incorrect the entire pipeline fails. Document intelligence also enabled the information like the weight of the text, which was needed by the client
Computer vision was required to manipulate the detection of various requirements e.g. the font size of the text on the label, the presence of kosher symbol. It was also used to crop the label from the pdf file where there was additional information other then the label.
AI-based algorithm was for pointing out the corrections in the text segments on the label. According to the client’s requirement, all the text segments had to be of a specific structure with certain keywords, this was enabled through AI.
Development Structure
We began the orchestration process by building a pipeline, the data in the form of a change order (EAW number) comes from the client’s database as soon a label is brought in the compliance. The PDF associated with this EAW is downloaded and the artworks associated are fetched as well from the client’s database.
This PDF is then cropped to only extract the label, which is then passed to the OCR for text segmentation and extraction and for the detection of font weight. The extracted segment coordinates are used to extract the independent alphabets in each text segments which are then used for computation of the font size. After this the spell check is performed on all the extracted text.
After this the text validations are applied on each segment, this includes raising a validation is a statement is incorrect, if it is missing, if it should not be on the label. Kosher symbol is detected using the cropped pdf (which was cropped in the second step). The validations related to the kosher symbol are also like the regular text validations.
All these validations are complied and posted on the comments section in the product management dashboard designed by the client under the eaw number. An excel workbook which includes all the comments along with there headers is attached within the dashboard as well. Once the process is completed the AI processed flag in the dashboard is checked as well.
The logs of the process along with the cropped PDF of the label are sent to the BLOB storage for further analysis. This pipeline is hosted on the Azure VM using Fast API and utilizes Azure Communication Services for providing push notifications in case of error within the pipeline.
Overall Architecture of the Solution
The architecture represents a AI-powered label/film processing pipeline hosted on Microsoft Azure. The architecture includes multiple stages, from data ingestion to validation and monitoring, ensuring compliance with business requirements. Below is a breakdown of the key components and workflow:
- Labels and films are ingested via an online label/API PDF document retrieval system.
- The system accesses EAW (Engineering Action Work) numbers with compliance review status and artwork attributes.
- The labels/films are passed to the browser for processing.
- Azure AI Document Intelligence extracts text from labels/films.
- Custom models trained on Azure Form Recognizer identify and segment text accurately.
- Extracted text is then passed for further validation.
- Font size is computed using bounding box dimensions from the extracted text.
- Text weight metadata (bold, italic, color characteristics) is also extracted.
- A custom vision model detects the Kosher symbol on labels using Azure Custom Vision Services.
- The result is passed for validation against label requirements.
- OCR results are analyzed for validation.
- Regex-based validation ensures that each text segment meets label requirements.
- Spell checker identifies misspelled words and suggests corrections.
- Validation outcomes determine whether the label passes compliance checks.
- The FastAPI framework is deployed on an Azure Virtual Machine to process validation rules.
- The backend application handles requests and responses between services.
- Validation results are sent to the dashboard
Challenges & Resolutions
In addition to the initial challenge of adapting OCR technology to diverse label designs, integrating new solutions with the client’s established systems without causing disruptions was a significant challenge. Another major hurdle was developing an effective method for cropping labels accurately from PDFs, given the variability in label sizes and placements.
We enhanced the OCR algorithms to improve recognition accuracy across different label types. To ensure smooth integration, we adopted a phased approach and conducted extensive training sessions for the client’s staff. For the cropping issue, we utilized sophisticated algorithms in OpenCV to detect and accurately segment labels, even from highly cluttered or noisy backgrounds
Key Takeaways
The automated label validation solution delivered a significant transformation for the client:
- 45% Increase in Cost Savings: Automation reduced the reliance on manual validation, directly lowering labor costs and minimizing the need for rework due to errors.
- 300% Faster Label Validation: The process to analyze and validate labels was significantly reduced, allowing the client to gain actionable understanding with much faster speed.
- 30% Increase in Productivity: By automating the label validation process, the client was able to reassign resources to other critical tasks, resulting in a noticeable productivity improvement.
- Seamless integration of Azure AI Services for label verification.
- Advanced image processing techniques for accurate text validation.
- Scalable cloud-based architecture, ensuring high availability.
The solution has successfully transformed label/film validation more accurate and faster, replacing a heavy manual process with an efficient AI-based solution. The solution ensures high accuracy, regulatory compliance, and operational productivity by using Azure cloud services.
Author
Zeeshan Mukhtar
Websites used to be something you built once and basically forgot about. That doesn’t work …Read More »- Learn how to plan an Optimizely CMS 13 upgrade with .NET 10, Optimizely Graph, Visual …Read More »
- Learn how AI meeting notes automate summaries, action items, and insights from video meetings using …Read More »
