Many of our time’s most pressing issues, from clean energy to environmental justice, necessitate novel approaches to the craft of scientific experimentation. This is particularly evident in the field of electron microscopy. Researchers are increasingly inundated with data and constrained by traditional operating models as they use this powerful window to peer into the atomic machinery behind today’s technologies. If we are to make breakthrough discoveries, we must incorporate artificial intelligence and machine learning into our scientific instruments.
Steven R. Spurgeon’s forecast of the current and future state of microscopy and instrumentation in scientific experimentation. Spurgeon, a materials scientist at Pacific Northwest National Laboratory (PNNL), is a world expert in the use of electron microscopy to study nanomaterials. He and his colleagues at PNNL are reimagining the discovery and design of new material and chemical systems by incorporating cutting-edge computing and data analytics into instrumentation.
As a result, academia and industry are seeking solutions from these PNNL electron microscopy science experts. PNNL is at the forefront of thought leadership in this expanding research area, and it is now bringing advanced technologies to market in order to accelerate scientific discovery.
An evolution in scientific experimentation
Spurgeon and his colleagues are attempting to address a problem that is common across multiple industries: experts are swamped with large amounts of data and hampered by outdated operating models, making knowledge extraction difficult. All domains are grappling with this burden, from new battery development to emerging quantum computing technologies.
The opportunity for automation in instrumentation is clear in advanced manufacturing—modernization would have an immediate and transformative impact. Failure analysis is carried out on a massive scale in the semiconductor industry—24 hours a day, seven days a week. To ensure the quality and reliability of microelectronics, microscopes and other systems must screen millions of transistors. Experts are increasingly concerned with how to translate these massive data streams into quick and understandable decisions that reduce costs.
The solution they seek necessitates hardware and software architectures capable of simulating cognition in the human brain. This would allow for the evaluation of unique scenarios while leveraging computers’ ability to scale analysis to different types and volumes of data indefinitely.
Many of the greatest challenges of our time, from clean energy to environmental justice, require new approaches to the craft of scientific experimentation. This is exceedingly apparent in the field of electron microscopy.
Industry-driven technology transfer
A team co-led by Spurgeon shared its vision for electron microscopy infused with the latest advances in data science and artificial intelligence in an October 2020 Nature Materials commentary. In the present day, this vision is being realized within PNNL’s Radiological Microscopy Suite. Researchers there have created a prototype of a next-generation microscope platform, and industry players are taking notice.
To commercialize the application, PNNL and Japan Electron Optics Laboratory/Integrated Dynamic Electron Solutions (JEOL/IDES), a world leader in electron microscopy, recently signed a licensing and co-development agreement. They will work together to bring the platform’s core concept to market—using minimal, or ‘sparse,’ data analytics to perform image classification—an important step toward instrument automation. The technologies developed through this collaboration will be refined and made available to research organizations and private industry. By using the platform, these experts will be able to process microscopy data without the need for entirely new instrumentation hardware.
“JEOL/IDES sees a clear need for improvement in the acquisition and analysis of microscopy data. This does not simply refer to automated instruments, but to smart automated instruments capable of expertly and effectively acquiring data “said Tom Isabell, JEOL/vice IDES’s president of product management. “We need to create a new paradigm in which data is collected efficiently and vast amounts of data are intelligently analyzed, leading to an even more efficient way of collecting additional data. PNNL has demonstrated global leadership in adopting this smart microscopy model, and JEOL/IDES looks forward to working with PNNL to develop and implement these new technologies.”
The broad application of this platform reflects the intent of PNNL’s Office of Technology Deployment and Outreach, as well as Jennifer Lee’s early work in spearheading the licensing agreement with JEOL/IDES. Her interactions with industry partners inspired her, as she heard a consistent theme: the labor-intensive, manual work involved in processing large volumes of microscopic data was simply too onerous. Industry partners were looking for multi-faceted expertise, not only in materials and electron microscopy science but especially in data science, which is inherent in a research entity that can deliver a solution quickly.
“At PNNL, we approach all of our technology transfer efforts from an industry standpoint. We work hard to understand the industry’s concerns and bring them back to our scientists for resolution “Lee stated. “For example, in our work with JEOL/IDES, there was immediate support and palpable expertise for developing an approach that could replicate the human brain’s decision-making capabilities, resulting in the quickest laboratory-directed commercialization effort, from start to finish, yet.”
Automation meets electron microscopy
The next-generation microscope platform developed by PNNL employs a never-before-seen analytics and control architecture. To extract statistical information quickly, experts are redesigning the electron microscope’s foundation, leveraging low-level system automation, domain-grounded data pre-processing, and emerging sparse data analytics. They’re making significant progress toward the microscope of the future, which will be highly integrated and automated and capable of tackling problems in energy storage, quantum information science, and other fields.
“Steven and his colleagues are working on a long-standing issue in the control and operation of electron microscopes. Their approach has the potential to have a significant impact on the scientific community by assisting researchers in conducting richer and more efficient large-scale analyses “Sergei V. Kalinin, a corporate fellow at Oak Ridge National Laboratory and a pioneer in machine learning and automated experiments in electron and scanning probe microscopy who was not involved in this study, explains.
Spurgeon assembled a team from within and outside PNNL to bring the microscopy platform to life, including fellow materials scientist Matthew Olszta, statistician Sarah Akers, computer scientist Derek Hopkins, and Kevin Fiedler, a mathematician from Washington State University. Spurgeon and Olszta’s expertise in microscopy was a perfect match for Akers’ few-shot machine learning, which represents a new type of data analytics that can make decisions based on very few examples. Spurgeon enlisted the help of Hopkins, who specializes in hardware/software integration and lab automation, to create a centralized instrument controller.
Hopkins and Fiedler created an architecture for processing and analyzing incoming images, allowing for large-area montaging and stage feedback. The team’s machine learning work is currently being reviewed in an article led by Akers titled “Rapid and Flexible Segmentation of Electron Microscopy Data Using Few-Shot Machine Learning,” with a more detailed article on the system to follow. Spurgeon is also working on a number of joint appointments.
The prototype microscopy system is now being used at PNNL on two flagship transmission electron microscopes, a JEOL GrandARM 300F and a JEOL ARM 200CF, with the goal of expanding it to other instruments in the future. This one-of-a-kind capability will allow for richer, round-the-clock statistical analysis that will take advantage of the laboratory’s best-in-class instrumentation.
Democratizing data-driven analysis
“The true potential of this work is that it can be extended to many other areas, leveraging PNNL’s expertise across multiple scientific disciplines,” Spurgeon said. “We have the opportunity to shift the discussion away from simply purchasing more powerful instruments and toward more informed modes of operation and analysis. Consider it the democratization of best-in-class analysis capabilities.”
To help accelerate this transition and support STEM workforce development, the PNNL team recently advised a group of students enrolled in the University of Washington’s Data-Intensive Research Enabling Clean Technologies (DIRECT) capstone program. The students were given the task of creating a graphical user interface for interacting with the few-shot model. This web-based application enables end users to process their data in an intuitive manner and export the results for future use. The students finished a paper, released their code, and will present a poster at the Microscopy and Microanalysis Virtual Meeting in early August.
On the road to the future
In addition to the team’s publications and the licensing agreement, other upcoming activities, such as an invited tutorial and four talks planned for the Microscopy & Microanalysis Virtual Meeting, demonstrate widespread enthusiasm for the microscopy platform. The annual meeting, hosted by the Microscopy Society of America, is open to its 3,000 members and is regarded as the premier event covering original microscopy research.
These activities, taken together, help to spread the current and future potential of this new platform. This will enable large-scale experimentation and the development of richer, more meaningful physical models for technologically relevant systems. The team’s work is just getting started as they plan for the system’s full implementation and expand on their machine learning work to increase the power and generalizability of their approach.
Spurgeon concluded, “We began with a new approach to data classification in the microscope, but we’ve expanded to address how we as a community approach experimentation. Traditional approaches are very manual and labor-intensive, but most importantly, they are incapable of keeping up with the latest hardware generation. Our platform, we believe, is the first step in that direction. The response from the scientific community and industry has been extremely positive, which is extremely gratifying.”
