Dealing with Verification Data Overload (16:25 - 17:15 on Wednesday)

Lan XiaProductivity Squad
Lan LambertProductivity Squad

Massive amounts of test logs and console output are generated each day at the AdoptOpenJDK and Eclipse OpenJ9 projects because of the huge number of tests multiplied by the number of versions, platforms and implementations tested. This ‘noise’ from daily, personal, and pull request builds requires processing in order to consume and make sense of it (more than 6G of test output is produced daily from Eclipse OpenJ9 test jobs alone, much more from AdoptOpenJDK). In some cases, we can instrument the test code to be less verbose. In all cases, we must take this raw data and refine it to more easily and effectively understand what next action to take. This presentation covers our approaches to handling the copious amounts of verification data, in order to make it meaningful and to give guidance on next actions. From a quality assurance perspective, there is value in results summary and aggregation. We need best practices in the application of data visualization techniques, filtering, and categorization. We need to continuously evolve and improve, employing relevant technologies such as deep learning. Ultimately, the goal of our data refinery efforts needs to be to display the results efficiently, allow users to quickly narrow down the problems and better monitor the farm to support the complex continuous delivery pipelines.



Lan Xia is a team lead in Functional Verification Team in IBM Runtime Technologies. As a software developer for over 10 years, she has extensive experience in software development, web development and test management. She is a committer on the open source project - Eclipse OpenJ9. And she is also an active contributor for Eclipse OpenJ9, AdoptOpenJDK and Eclipse OMR project. She creates and provides support for test frameworks, test infrastructure, CI test pipelines, various testing and node.js based test result summary service. She is self-motivated and constantly experimenting with new technologies and techniques to assure better testing.

Shelley Lambert is a Test Lead for IBM Runtime Technologies team. Over the course of the last few years, they have systematically shifted from a very inward-focused test and development team to open-sourcing all of their product and test code into Eclipse OMR and Eclipse OpenJ9 projects, as well as their delivery pipelines to the AdoptOpenJDK project. Her team has delivered the test strategy, test code base, test frameworks and delivery mechanisms into these 3 projects. She is a committer at Eclipse OpenJ9 and at AdoptOpenJDK, where she also serves on the Technical Steering Committee. She has presented at JavaOne and Eurostar conferences and many smaller meet-ups on a variety of topics, drawing stories and lessons from her experiences in the 3 open projects in which she is most active.