Enabling scalable data streaming for R&D innovation

Challenge: Integrating R&D test data efficiently

Our client’s R&D test labs generate valuable information to drive product and process improvements. They needed an efficient way to collect and transfer results from multiple test machines to central data systems for analysis and ongoing research. 

Their existing Ignition SCADA setup, while important for daily test operations, did not offer a built-in method to stream data to Kafka. This made it difficult to scale data integration for future R&D projects. The goal was to find a practical, admin-friendly solution focused on R&D needs – one that would not require changes to live production environments.

Solution: Custom Ignition module for test data streaming in R&D 

To support R&D teams with reliable access to test results, Proekspert piloted a custom module for Ignition SCADA, specifically designed for test environments. The module connects multiple lab machines to a central Kafka system using admin-configurable streaming. Integration with Ignition leverages the platform’s tag-based architecture – by subscribing to tag change events, the module detects updated test data across a variety of lab devices. 

Configuration is managed through an admin-facing interface, built using Apache Wicket, that allows lab specialists to set up Kafka connection parameters, define topics, and adjust critical policies (security, retries, timeouts) without needing specialist coding skills. Rather than introducing new data formats, the pilot made use of the client’s existing Avro serialization, working with their team to ensure seamless compatibility with downstream analytics platforms and Confluent’s Schema Registry.

Results: Proof-of-concept findings and technical learnings 

The proof-of-concept demonstrated how custom data streaming from R&D lab devices to Kafka can be implemented and tested within the client’s environment. 

Specifically, the project: 

• Demonstrated the ability to stream test data to Kafka from lab devices using a custom Ignition module. 

• Verified compatibility with the client’s established Avro serialization and downstream analytics tools. 

• Piloted an admin-configurable setup for managing Kafka connections, topics, and operational policies, tailored for R&D administrators. 

• Evaluated the adaptability of integration patterns from other departments, identifying specific adjustments needed for R&D workflows. 

The project delivered hands-on technical knowledge and clarified what must be refined before any broader deployment in other labs. The system is not in production; further adaptation will guide the next phase.

Impact for the client’s R&D organization 

This proof-of-concept strengthened the client’s understanding of how to approach scalable data integration in their R&D labs. The project validated technical concepts for streaming test data from lab equipment to central systems, provided hands-on experience for R&D administrators configuring and managing the solution, and highlighted practical considerations for adapting these methods to lab workflows. 

As a result, the client now has: 

• A proven integration approach that can inform future expansion of test data streaming and analytics in R&D. 

• Clear insights into necessary adaptations to lab systems and configuration approaches before broader deployment. 

• A pathway for connecting additional lab equipment in the future, using familiar, admin-friendly technologies and integration patterns. 

This PoC provides a foundation for informed decision-making as the client evaluates next steps for digitalization, integration, and expansion of lab data capabilities within their R&D operations. 

Technologies

Java, Gradle, Ignition SDK, Apache Kafka, Avro, Apache Wicket, Confluent