Ground-based autonomous vehicles like robotic vacuum cleaners and lawnmowers are already among us in the consumer space. To perform the tasks we ask of them, these robots use simplified autonomy stacks — the layers of technology needed for perception, navigation, control and decision-making.
Now imagine the complexity involved in developing autonomy stacks for a rugged 6×6 off-road vehicle that conducts search-and-rescue missions, mine-clearing operations, firefighting, and surveillance and reconnaissance — all taking place in a dangerous and changing environment. This is exactly why scientists, engineers and roboticists at The University of Texas at Austin and the U.S. Department of Defense are collaborating to mature technology that helps keep military personnel safe and minimizes human risks in hazardous situations.
Since 2019, Army Futures Command’s collaboration with TEXAS Robotics has led to a novel method for rapid adaptation and integration of robotics and AI systems research. This innovative model is quickly advancing transformative research into early-stage U.S. Army prototype software and vehicles and has become an example of how basic university research is transferred to the military. The ongoing effort also allows UT faculty and students to use and expand their research skills on problems the Army must solve in critical technology areas.
“I’m proud to see how our collaboration with the Army has not only accelerated AI research advancements but also been a catalyst in the expansion of full-service robotics programming at UT,” says Dan Jaffe, vice president for research, scholarship and creative endeavors. “It complements the University’s impactful work in service to society and greatly benefits our students, who receive training to be leaders in the many emerging fields working in artificial intelligence and autonomous systems.”
In this new defense research paradigm, the Army evaluates the progress made by researchers earlier in the technology development process. In turn, UT researchers take this feedback and can more quickly input it into their own technical processes, so that advancements happen at a more rapid pace. The increased efficiency results in cost savings for the Army and allows for more “soldier touch-points” to ensure that new technologies are trusted and embraced by the end-user. Students are also engaged at every step in this model, providing unparalleled learning experiences.
Integrating Research Across UT Into New Systems
The creation of the Robotics Center of Excellence, which opened in 2020, has been essential to the evolution of this mutually beneficial workflow. Located at UT and funded by the Army, the center encompasses all major robotics research disciplines at the University and is designed to enable highly collaborative work and support a broad range of industrial applications.
Faculty and researchers from different departments across campus with relevant expertise in AI, vision, control, vehicle design, simulation, power, software/network infrastructure, perception, data analytics, system design and sensors are now able to come together to streamline the integration of complex new systems and technologies.
Two UT faculty members have been instrumental in reshaping the research infrastructure in autonomy and robotics at the University and also have created a pathway for other visionary researchers and students:
Computer Science professor Peter Stone and Mitch Pryor, a mechanical engineering research professor, both supervise cutting-edge autonomous systems research by students, post-docs and research staff. They direct Texas Robotics and the Robotics Center of Excellence, respectively, and have served as principal investigators on multiple projects involving the Department of Defense, Department of Energy, and many others.
One example of a successful technology transition has been research on human-in-the-loop learning from robot navigation, specifically the Adaptive Planner Parameter Learning (APPL) algorithm, a paradigm that devises a machine-learning component on top of existing navigation systems and allows for dynamic adjustment of parameters to adapt to different regions. Originally developed and prototyped in Stone’s lab, APPL was later transitioned onto platforms for the DEVCOM Army Research Laboratory (ARL) and Ground Vehicle Systems Center by Pryor’s and Stone’s research groups.
Stone is a prominent AI expert on machine learning and dynamic environments, focused on expanding knowledge in long-term autonomy and human-robot interaction. He teaches computer science courses and is a founding member of Good Systems, a campus-wide research grand challenge initiative focused on ethical AI.
Pryor is an authority in applied robotics and automation. He is also the co-founder of the Nuclear and Applied Robotics Group and has been teaching undergraduate and graduate courses in mechanical and electrical engineering on the Forty Acres for two decades. His research focuses on supervised autonomy for robotics and automated systems through advanced control, decision-making and sensor integration in hazardous environments.
“As an interdisciplinary group, Texas Robotics can explore a lot of synergies across a lot of different perspectives,” Stone says. “The military partnership inspires research and creates a path to impact. We’ve learned that embedding Army technical experts in the research process from the beginning makes it more efficient and effective.”
The process is also enabling new ideas that are earning broader attention in scientific and engineering communities. UT researchers and affiliate partners are presenting their findings on new paradigms and sharing insights and knowledge at professional conferences and publishing papers such as “APPLE: Adaptive Planner Parameter Learning from Evaluative Feedback” in IEEE Xplore journal.
This collaborative paper authored by UT researchers and Army and industry scientists represents a less-demanding modality of user interaction based on their APPL paradigm in which people without expert-level knowledge in robotics can help train and improve autonomous vehicle navigation in a variety of environments. Stone and several of his students, along with DEVCOM Army Research Laboratory (ARL) research scientist Garrett Warnell, gave a robot the ability to learn how to autonomously navigate by observing and imitating a human demonstration of navigation.
“Our partnership serves as a strong example of how deep collaboration between academia and the Army can work to the benefit of our nation,” Warnell says. “The Army is one of the nation’s largest and most important institutions, so the potential for impact is huge.”
Pryor’s integration prototype research and work with Stone on human-in-the loop machine learning for autonomous navigation has developed new capabilities that quickly can be assessed and integrated into Army software or specific platforms for testing and evaluation by the Army.
“The Robotics Center of Excellence takes great foundational robotics work and accelerates the transition of that technology to the military,” Pryor says. “We do this by evaluating the systems on Army-relevant platforms and addressing research issues that arise when transitioning from the University to real-world applications. UT has become an example of how basic university research is transferred to military.”
This year, UT Austin was awarded five new agreements from the U.S. Army to grow defense research capabilities. Each of the awards is a continuation of projects that began under the original cooperative agreement.
Researchers are also contributing their significant expertise in mobility and motion-planning, core components of autonomous systems, across military services and federal funding agencies. Multiple Texas Robotics faculty and students are participating in a groundbreaking U.S. Space Force initiative that will propel innovative technologies for in-space operations. Spacecraft proximity operations, in-orbit repairs, and refurbishments are central to the goals of this effort to significantly expand the range and scope of space missions with safety and operational reliability guarantees.
Advancing Technology Across Land, Air and Space
The technological leaps UT researchers are making happen faster with Army scientists onsite and working closely with the research labs. Warnell specializes in AI and machine learning and is also a visiting researcher-scholar within the Department of Computer Science, where he is involved in all aspects of the research — from meeting with students to designing new algorithms, debugging code and scoping experiments, to coauthoring papers and serving on dissertation committees.
Similarly, joint UT-Army researcher Christian Ellis contributes his engineering expertise to build safe and robust learning-enabled, off-road ground systems — primarily by coauthoring publications aimed at training students to address fundamental challenges when developing real-world autonomous systems.
“We at the Army get access to the world-class research talent that UT attracts, and we’re able to learn from and help shape the science happening at the fundamental level,” Warnell says. “Meanwhile, UT researchers have the opportunity to work with us on some of the most challenging and fun problems in their field, and, more importantly, have their work help inform the Army’s understanding of a particular scientific area.”
Aerospace Engineering professor Ufuk Topcu, Ellis, and associated research staff at the UT Center for Autonomy contribute to Department of Defense research by providing theoretical and applied research-enabling robots to perform safe autonomous navigation in complex and dynamic environments that are unknown before operation.
In a collaborative agreement between research partners, they began working toward this goal in 2020, successfully completing two projects. The initial research resolved a gap between what a remote human operator knows and what the robot detects by modeling the robot’s limited sensor range, perception capability and associated uncertainty.
Secondly, a robot learning framework was introduced to break down tasks into smaller, manageable components that can be learned one by one and then combined to solve more complicated problems. Specifically, they provided a test and evaluation framework in which robots are quickly trained in a simplified virtual environment, evaluated in a high-fidelity simulator that closely replicates the real world, and then deployed on real world autonomous hardware. These two research thrusts provided software products integrated within ARL’s autonomy simulator, and real-world hardware – demonstrating the ability to solve Army level problems where limited sensor range in unknown environments is common.
Maruthi Akella, a professor in the Department of Aerospace Engineering and Engineering Mechanics, and his research team are providing the algorithmic foundations for how multiple air and ground robotic vehicles can coordinate among themselves by dividing a large task into individual sub-tasks with minimal human oversight. Under this framework, vehicles could be dynamically and seamlessly introduced or removed during a mission to complete objectives.
This coordination requires overcoming a series of computational challenges involving multiple variables such as size-weight-power constraints, robotic hardware failures and outages due to rugged environments, decentralization requirements, and reliance upon limited and uncertain communications networks.
Akella’s research team bridges three critical areas in autonomous systems: trust, information fusion, and how tasks for the vehicle fleet are negotiated and distributed over large-scale networks. Initial implementations of these new algorithms were reported at the 2024 American Control Conference and shown to scale favorably with other vehicles while still meeting Department of Defense budget requirements for numbers of vehicles and communication bandwidths.
Developing Talent in AI: Student Training Through Research
From the beginning of the UT-Army partnership, students have been working on applied research in robotics and autonomous systems alongside faculty and Army technicians, enhancing their preparation for careers in emerging areas such as robotics, AI and machine learning, quantum computing and autonomous systems. This direct training through research prepares students to thrive in the workplace’s emerging interdisciplinary spaces. To date, over 150 undergraduate and graduate students have worked on Army-funded research projects.
Graduate students are problem-solving alongside faculty and AI experts from the military and industry, learning as they go on state-of-the-art technology. They are involved in every step of demonstration and evaluation and gaining experience in critically important areas. Their research findings are also being accepted in peer-reviewed journals and they are sharing research at prestigious conferences such as IRO5 and ICRA.
Christina Petlowany is a mechanical engineering doctoral candidate working with Pryor and was also appointed to the ARL Research Associateship Program.
“I’m passionate about using robots to prioritize human safety,” Petlowany says. “In UT’s Nuclear and Applied Robotics group, a core research thread of ours is placing robots in precarious environments to keep people removed from the dangers.”
She is researching means of human-robot interaction and robotic control to make teamwork learning between people and robots more manageable. The goal of her work is to gain insights into how people process visual information in various mental or emotional states and how it correlates with robot performance level. This analysis is especially useful in fine tuning human-machine interaction and helps to adjust a robot’s behavior in response. For human-machine teaming to work, robots must be taught to understand the nuances of human physiological and psychological states.
“My research is relevant to robotic domains beyond defense,” Petlowany says. “As robots become more commonplace, it is necessary to create manageable systems and meaningful ways for people to utilize those systems. This applies to self-driving cars, delivery, home service and disaster aid robots and automated factory systems.”
Academic opportunities in robotics and autonomous systems have also grown for students. At UT, there is now one of the first undergraduate robotics program in the country that allows high school students to apply directly as part of their freshman admissions application. Also, undergraduate students who wish to develop proficiencies in robotics while pursuing a major in a related field can earn a minor in robotics or they can apply to research assistant positions offered by current Texas Robotics graduate students.
Texas Robotics now offers a Graduate Portfolio program where graduate students can earn a “Certificate of Expertise” distinction on their master’s- or doctorate-level degree transcript.
“I am very proud of the number of students that have come through our programs and are now working in the areas that originally funded their projects,” Pryor says. “Our graduates are using their studies and research experiences to develop new and exciting technology. Their findings are turning into applicable, life-changing work that will continue outside the University.”
