Past CoE-funded Projects

PI Researcher: Chenliang Xu

In robot manipulation, adapting to dynamic environments with flexible task specifications is challenging. Language-based vision manipulation systems offer a solution by linking language instructions to visual data and generating actions. However, current approaches often develop vision models and action policies separately, leading to poor integration. To address this, we propose ACTLLM, a method that unifies visual interpretation and policy learning using large language models (LLMs). By generating structured scene descriptions and incorporating an action consistency loss, ACTLLM is expected to enhance the fusion of visual and policy elements, facilitating the efficient execution of complex tasks within a multi-turn visual dialogue framework.

PI Researcher: Michael Heilemann

This project, in conjunction with ADVIS Inc. is focused on developing a device to cost-effectively bring machine health monitoring to a broad spectrum of Department of Defense (DoD) assets (vehicles, pumps, etc.), where the implementation of conventional monitoring systems is cost prohibitive. To meet size (~1 in.3) and power consumption (battery life of ~3 years) requirements, the device utilizes low-power embedded machine learning (ML) models trained on data acquired by a vibration sensor. Spectral features extracted from a recorded signal are used to train an embedded ML model to perform tasks such as the detection of anomalies and faults in mechanical systems.

PI Researcher: Mark Bocko

This research will support new Rochester startup, Obscure Signals, that is developing AI-based tools for the preservation and restoration of historical audio recordings. Inference of the signal processing steps employed in historical recording methods currently is a time-consuming process of experimentation and expert listening assessment. Historical recording methods are also intrinsically lossy, for example the original audio signal may be compressed. Thus there is a need to employ generative AI models in the restoration process. The research that will be conducted in this project will explore the use of “neural optimal transport” for such generative AI tools.

PI Researcher: Jaime Cardenas

We propose to develop a novel fiber array to chip alignment system using deep learning and computer vision for automating the alignment process for a fiber array.

PI  Researchers: Tong Geng

During the first year of CoE support, PI has demonstrated that Ising Machine with proper augmentation can be used to efficiently solve simple graph learning problems and outperforms GNNs in terms of inference speed and accuarcy. The resulting new graph learning paradigm is dubbed “IML”. This year, this project aims to improve IML through co-design of Hamiltonian, hardware architecture, and training algorithm and develop “DS-GL”, that effectively harnesses nature’s power within flexible, self-trainable, real-valued hardware Dynamical Systems to solve real-world Graph Learning problems. DS-GL aims to achieve fast training, high accuracy, and low inference response samultaneously.

PI Researcher: Wayne Knox

We have made significant progress in imaging nerves using polarized light sensing at different wavelengths. Starting in the visible spectral range and using raw or frozen chicken and now progressing to human cadaver nerve imaging, we have developed new methodologies of image processing to highlight the nerves to stand out from the surrounding tissues. We propose to expand our project to include a FIH (First in Human) surgical study which will present new data science challenges with image motion, and also to push into the near infrared for better tissue penetration, and to develop an endoscopic version of our technology.

PI Researcher: Wayne Knox

Development of a new Intraoperative Instrument Using AR/VR technology: We propose to continue and expand our recently started research project on using multispectral polarization technology for the detection of nerves and to learn how to incorporate it into a VR/AR/MR technology as a new surgical aid. This device could help to avoid accidental nerve cuts during routine surgery. The project will develop a research prototype for use in live human surgical clinical trials. We will test different modes of image processing such as artificial intelligence and methods of data science to determine the best modes of operation according to the surgeon’s requirements.

PI  Researchers:
Michael Huang (left)
Tong Geng (right)

Ising Machine Learning: When the Ising Model and the Ising Machine Meet Machine Learning: The drastically increasing computational cost of Machine Learning (ML) has brought concerns on environmental sustainability and challenges in its practical deployment. Inspired by the recent success of using the Ising model and the Ising machines to solve NP optimization problems with orders-of magnitude speedup and energy reduction, this project will investigate how to leverage the Ising model’s strong expressivity and the Ising machine’s extraordinary computational speed in learning problems to create a new ML paradigm, Ising Machine Learning (IML). IML has the potential to enable the sustainable development of ML by significantly reducing the computational cost of current ML modelities with increased accuracy.

PI Researcher: Pablo Postigo

Optical filters are one of the key products in optics. Nevertheless, there are still some limitations that affect the optical performance of optical filters, like angle of incidence or the operative spectral ranges. Pablo Postigo and IDEX Health Science, LLC conducted research to develop computational methods based in AI to design new optical filters with enhanced properties, like higher resistance to the angle of incidence, a wider transmission/ reflection band or a higher number of spectral bands. Furthermore, the use of a filter design procedure based on AI may open new paths for the development of filters with other enhanced properties like polarization selectivity.

PI Researcher: Andreas Savakis

Domain Adaptation using Vision Transformers: Domain adaptation aims to overcome the distribution shift between the target domain used for testing and the source domain used for training. We propose to utilize Vision Transformers as an alternative to convolutional features for fine-grained domain adaptation. Vision transformers, inspired by natural language processing, offer excellent performance due to their attention mechanism, and are promising for fine-grained classification. We propose to incorporate vision transformers in the source hypothesis transfer framework and apply our fine-grained adaptation method on the rare planes dataset, which includes synthetic data for training and measured data for testing.

PI Researcher: Chunlei Guo 

Neural Network assisted Femtosecond Laser Fabrication of Anti-bacterial Surfaces: The PI’s lab has been working with Pfizer in developing materials for biomedical applications. Pfizer plans to launch a new project in developing anti-bacterial surfaces to reduce the stringent requirements for equipment and vaccine container sterilization. Pfizer is a leading producer of COVID-19 vaccines. The economic values of solving these issues are immeasurable. We plan to incorporate our recently developed anti-bacterial surfaces to alleviate the aforementioned issues. The experimental procedure of producing anti-bacterial surfaces is complex and time-consuming. In this project, we will develop a neural network and genetic algorithm to optimize and speed up these fabrication processes.

PI Researcher:
Ling Wang

This project aims to develop a label-free and automated cell classification method for pharmaceutical research using phase imaging and deep learning techniques. The proposed approach leverages the strengths of multiple deep learning models and ensemble learning techniques to improve the accuracy and efficiency of cell classification. The project targets phase contrast images, which are label-free and provide detailed information about cell morphology and dynamics. The proposed method has the potential to benefit pharmaceutical research by providing an accurate and efficient tool for cell classification, aiding in drug discovery, toxicity testing, and other aspects of pharmaceutical research that require precise characterization of cell types and behaviors.

PI Researcher: Yunbo Zhang

This project aims to extract tacit knowledge from experienced machinists and transfer it to the next generation of machinists. We will collect multimodal data from experienced machinists through an experimental set for tool wear determination in machining; develop machining Learning methods for tacit knowledge extraction consist of: behavior data fusion based on an integrated Bayesian hierarchical and infinite hidden Markov model, and identifying useful tacit knowledge patterns; and integrate tacit knowledge into the XR system. The project’s ultimate goal is to transfer the developed XR training system to our industry partner, AA Tech, for their training of new machinists.

Causal modeling represents an active area of research for determining causal relationships when a randomized controlled experiment is not available. The RPI team has developed extensive experience building causal models using micro-level data. Jason Kuruzovich (left) and Kristin Bennett (right) will be worked with Velan Studios to apply methods to the context of answering business relevant questions on engagement outcomes in gaming that are also relevant for theory development. Questions may involve training, skill matching, in a skill-based collaborative / competitive game Knockout City. The analytics capabilities will be further incorporated as infrastructure for future games launched by Velan Studios and available to startups funded through Velan Ventures.

AI algorithms, especially deep learning, learn biases from data; thus, it is urgent and vital to identify and mitigate biases in these algorithms. However, previous bias identification pipelines rely on human experts to conjecture potential biases (e.g., gender, color, etc.). This approach might work for limited domains and problems but have no chance to scale when considering general AI deployment; humans alone cannot realize all the biased attributes, e.g., identifying biases for a bridge classifier trained on ImageNet. Kitware, Inc. has contacted Chenliang Xu, Associate Professor in the Department of Computer Science and Wilmot Distingshied Professor 2021-2023 at the University of Rochester, to develop novel, scalable human-machine collaborative systems to assist humans in discovering hidden biases in any image classifiers.

Dr. Michael Huang (right) and Dr. Zeljko Ignjatovic (left) worked with L3Harris to explore two scalable architectures (a microchip and a multi-processor types) based on a previously proposed CMOS-compatible Ising machine will be explored and characterize to address issues using an Ising machine.

Yufeng Huang (left) and Mitchell Lovett (right) worked with Monro Inc. will study how to help salesforce provide better customized discount to consumers. 

Leveraging data from a large automobile service company, this project will use business analytics methods and develop a structural econometric model to characterize consumer demand and bargaining with salesperson. They will use the model to predict the market outcome under different interventions to salespersons, including setting a monthly discount quota, or completed forbidding discount.

Mark Bocko, Distinguished Professor, Electrical and Computer Engineering
Professor, Physics and Astronomy
Director, Center for Emerging and Innovative Sciences (CEIS) at University of Rochester, is working with Rochester based Immersitech, Inc. to develop a framework to evaluate the performance of spatial sound reproduction. This research combines representations of human peripheral auditory system response and low-level processing of binaural data in the brainstem with learning networks to infer auditory scenes from given acoustic stimuli.  The framework provides a tool to evaluate and guide the development of spatial sound reproduction systems.

Jannick Rolland will be working with LighTopTech Corp. to develop a microscope for Gabor-Domain Optical Coherence Microscopy (GDOCM) and related image processing tools will be developed in this project, enabling commercialization of pre-clinical and clinical GD-OCM for brain tissue and organoid characterization.

Dr. Huang (left) and Dr. Ignjatovic (right) from the University of Rochester are working with AMD researching Architectural Support to Increase Ising Machine Capabilities.The goal of the project is to develop advances in Ising machine technologies that will bring them closer to realizing their envisioned potential of orders-of-magnitude improvements in speed and energy efficiency when used to solve hard optimization problems (such as SAT). A concrete technical goal is to move from usual quadratic interactions to also account for qubic interactions that can facilitate mapping (and then solving) a broader class of problems more efficiently to the proposed Ising machine substrate.

Gonzalo Mateos, Associate Professor with the Dept. of Electrical and Computer Engineering, University of Rochester, is working with IBM on Learning to Localize Sources of Network Diffusion.