Abstract

In today’s literature there exists a gap within data quality research. Most devisable data quality dimensions are already explored. However, the dimension synchronicity which sets demands to data points from multiple time series to be sampled synchronously is not investigated to this date. This work comes up with a metric measuring synchronicity for a given set of time series and gives ways to improve data affected by bad synchronicity. The need for a visualization of metric results is explained. Consequently, a dashboard app is designed and implemented to visualize the developed data quality metric. A proof-of-concept test is conducted at an Industrie 4.0 demonstrator showing that the prototype can handle real-life scenarios. Two separate user studies test the dashboard for usability. The first uses qualitative research methods and is used to gather first feedback which is used to improve the design. The second user study is used to evaluate the usability of the dashboard. It uses quantitative research methods and utilizes the System Usability Score as an evaluation instrument. The resulting SUS score at 77.1 points shows that the usability of the dashboard is rated between good and excellent.

Synchronicity Metric & Improving Synchronicity

Synchronicity as a data quality has demands for multiple time series. Samples of different time series need to be in sync with each other to ensure that measurements happened at the same time. A metric is developed which measures the degree to which data is in sync with each other. The basic idea is group data points that are closest to each other from each time series into a record if they are deemed to be in synch. This will result in two possible kinds of records. The first has as many data points per record as there are time series analysed. These are called complete records. The second kind encompasses records with fewer or only one sample. These are called incomplete records. This classification is used to compute the simple ratio of complete records divided by all records. This metric is per definition a direct indicator for bad synchronicity since it will go down if fewer points are in sync with each other and vice versa. However, this ratio will also be low if data is impaired by other data quality defects. If by chance one time series itself has missing records the ratio will go down which is however not an effect caused by bad synchronicity but by bad completeness. This means that the defined ratio is also impaired by completeness problems of single time series. To overcome this issue and to decide whether a synchronicity problem is affecting the data one has to rule out other possible causes like the aforementioned completeness problems. The proposed solution does this by checking if all time series have even sampling times. A dedicated formula is used as a threshold in that regard.

After the synchronicity metric check a solution to improve data affected by bad synchronicity is proposed. It encompasses resampling and interpolation. The basic idea is to force every time series to be in sync with each other. First, a resample base is selected out of the affected time series. The remaining time series are then resampled accordingly and missing values are interpolated.

The following picture shows the whole process more detailed in a decision graph.

Visualization

With a synchronicity metric developed at hand, a dashboard is designed and implemented visualizing the process and metric results. The basic idea is to create a data-centric view where time-series graphs take up most of the screen. The following picture shows the finished prototype. Time-series graphs are stacked on top of each other sharing the same time axis. This is done to encourage the user to recognize matching data points since they are visually near to each other. On the right side, space is reserved for control elements and metric results.

Proof-of-Concept Test

To test the implemented prototype in a real-world scenario it is rolled out at the MyJoghurt plant located at the AIS chair. The following picture shows the set up used in the conducted proof-of-concept test.

Two sensor values from the plant are read through an OPC UA server and written into a local database. The dashboard is used to analyse the synchronicity of the sensor samples in two scenarios. In the first scenario, both sensors are sampled synchronously. The dashboard recognizes this property and shows "green" synchronicity as a result. In the second scenario, one sensor is sampled at a higher rate causing the samples of both sensors to be out of synch. The dashboard is also able to recognize this issue and gives out the expected response that the data is affected by synchronicity problems. This test proves the applicability of the implemented prototype in a simulated real-world scenario.

Two User Studies

The dashboard is also evaluated for its usability with hand-selected use cases. For this matter, two user studies are conducted. The first one is more of a qualitative nature. Three test participants are selected from the research assistants at AIS knowledgable on the topic of data quality. This study is used to acquire first feedback for the prototype. The study can be divided into three segments. First, the participant is introduced into the topic of data quality and synchronicity as well as how I solve the problem on how to measure synchronicity and how to visualize this process. Second, the participant is verbally given tasks they had to complete with the dashboard which cover its main features. Last but not least, a personal interview is conducted where the test participant is asked about their user experience when completing the tasks. The prototype is improved based on the feedback from the test participants.

The last study is of a qualitative nature and is used to evaluate the usability of the prototype. It is structured similarly to the previous study but now uses students as test participants. First, the student is introduced to the topic. Second, the student is given a task sheet with written tasks they have to complete with the tool. Lastly, the student evaluates the dashboard with a questionnaire. For the evaluation items from the System Usability Score (SUS) are used. Furthermore, the classification of the SUS into two subcategories, usability and learnability, is used. The following table shows the evaluation results in a table and a graph.

Using the adjective rating scale developed by Bangor, Kortum et. al. 2009 the aggregated SUS as well as the sub-score usability of the dashboard can be rated as good. However, the learnability score is a lot smaller achieving only an OK on the same adjective rating scale. This may be attributed to the fact that only one out of seven test participants stated that they already knew about data quality and synchronicity as a dimension of the same. The questions responsible for the learnability score ask about the participant's need of a technical person for using the dashboard and if they had to learn a lot before they could get going with the dashboard. Students may be influenced by their lack of subject matter knowledge to rate both questions with a negative response. The high dispersion of the score indicated by the standard deviation being more than a fifth of the total range of the score is influenced by the high increments of 12.5 the score is rated with. However, it might be possible that the prototype is simply lacking in the aspects of learnability.

Conclusion

A metric for evaluation of the synchronicity dimension of data quality is developed. It is extended by checks discerning it from other possible quality defects. This metric fills a predominant research gap in literature. Also, a dashboard visualizing and automating the metric is implemented. It is evaluated to possess good usability qualities. It helps users understand how well their data performs regarding demands to their synchronicity.

[PDF] 

[Slides Kickoff/Final]