Many emerging robotics applications require the use of multiple collaborating robots to operate under human supervision. To be useful in such applications, robots will need to act as smart assistants by automatically selecting actions, efficiently learning from the observed performance, safely operating in the presence of uncertainty, and appropriately calling for help during the execution of challenging tasks. Our group is making advances in physics-aware artificial intelligence that are being used to enable robots to automatically make decisions to meet aforementioned requirements.
Execution Monitoring, Learning, and Plan Refinement
Heuristics-based Multi-agent Task Allocation
Multi-Robot Task Allocation (MRTA) is a pre-requisite for many autonomous, multi-agent systems because of the need to intelligently assign tasks to teammates for maximizing efficiency. The focus of the research community has generally been on developing methods that optimize performance in relatively-controlled settings, assuming only a minor amount of uncertainty and possibility of failure. In many real-world missions, however, uncertainty and failure are paramount concerns because they naturally degrade performance, can have cascading effects on agents and future task allocations, and could prevent successful completion of a mission altogether. To overcome these challenges, we seek task allocations that enable resilient operations by considering more expressive models and improve the probability of mission success rather than strict optimization of some classical objective function, e.g., minimizing makespan. We are developing a novel heuristic for MRTA that removes common Markovian assumptions and incorporates state and uncertainty modeling in order to improve performance. When tested in a simulated search and rescue mission using real-world autonomous navigation data, our novel heuristic shows a considerable improvement in mission performance by modeling probabilistic task failure, agent failure, failure detection, and non-Markovian states.
Contingency-Aware Task Planning
Complex logistics support missions require execution of spatially separated information gathering and situational awareness tasks. Mobile robot teams can play an important role in the automated execution of these tasks to reduce mission completion time. Planning strategies for such missions must consider the formation of effective coalitions among available robots and assignment of tasks to robots with the goal of minimizing the expected mission completion time. The occurrence of unexpected situations that adversely interfere with the execution of the mission may require the execution of contingency tasks so that the originally planned tasks may proceed with minimal disruption. Initially reported potential contingency tasks may not always affect mission tasks owing to the uncertainty in the mission environment. When potential contingency tasks are reported, the planner updates its existing plan to minimize the expected mission completion time based on the probability of these contingency tasks impacting the mission, their impact on the mission and other task characteristics. To incorporate a contingency task in a mission plan, we require adding new tasks to the mission plan. A potential contingency task could be handled using two simple approaches. A naive conservative approach incorporates all reported potential contingency tasks in the mission plan without accounting for their probability to interrupt and adversely impact the nominal execution of the mission tasks. Alternatively, we can adopt a reactive just-in-time approach and only execute the contingency task when it becomes critical to the mission. For expected mission completion time minimization, both are suboptimal. The first approach might execute contingency tasks that had no impact on the mission. The second approach might create bottlenecks by keeping robots idle and waiting for the assigned robot to finish the contingency task. It would have been better to handle the contingency task earlier. We are developing a proactive approach to incorporate potential contingency tasks. We initially generate a plan for nominal mission execution. Detection of a new potential contingency task leads to replanning. The planner must update the mission execution plan based on the probability of the potential contingency task to impact the mission tasks. It may either immediately incorporate the contingency task in the mission plan or defer its incorporation until its information changes. We have demonstrated that the proactive approach to contingency task management outperforms both the conservative and reactive approaches.
Task-Agent Assignment for Mobile Manipulators
Multi-arm mobile manipulators can be represented as a combination of multiple robotic agents from the perspective of task-assignment and motion planning. Depending upon the task, agents might collaborate or work independently. Integrating motion planning with task-agent assignment is a computationally slow process as infeasible assignments can only be detected through expensive motion planning queries. We have developed three speed-up techniques for addressing this problem- (1) spatial constraint checking using conservative surrogates for motion planners, (2) instantiating symbolic conditions for pruning infeasible assignments, and (3) efficiently caching and reusing previously generated motion plans. We have shown that the developed method is useful for real-world operations that require complex interaction and coordination among high-DOF robotic agents.
Path Constrained Trajectory Planning
A tool needs to follow a path defined on the surface of a part to carry out this operation. The relative motion among the objects to carry out certain tasks can be defined as path-constraints. Robotic finishing, painting, inspection, fiber placement, etc are some of the operations where relative motion of objects can be encoded as path-constrains. The constrains include but are not limited to general end-effector constraints, avoiding collision and singularities, maintaining desired velocity and force at the tool-tip, and complying to the joint limits. These path-constraints or the motion of the objects can be generated using traditional motion planners. However, if we want to automate these tasks using robots, then we need to generate configuration space trajectories for robots in addition to generating the motion plans for the objects. The problem of generating such trajectories is known as path constraint trajectory generation problem. Traditionally, a version of the problem which imposes several path constraints is solved using graph search techniques. However, there are limitations of the approach such as discontinuities cannot be handled by the planner, methods to define path consistency are not included, and large problems cannot be solved. We are working on an approach where a reduced graph can be successively constructed by using fundamental rules which prune nodes and edges in the graph.
Point-to-Point Trajectory Planning
In many manufacturing applications, robotic manipulators need to operate in cluttered environments. Quickly finding high quality paths is very important in applications that require high part fix and frequent setup changes. We are developing a point-to-point path planning framework for manipulators operating in cluttered environments. It uses a bi-directional tree-search to find path and facilitates finding a balance between path quality and planning time.The framework dynamically switches between multiple search strategies based on the search progress to produce high-quality paths quickly. It builds on a previously developed sampling-based modular tree-search developed at CAM. Among other strategies, it includes a sampling strategy that can sample effectively in challenging regions of the search-space by using local approximations of the configuration space. It also includes an inter-tree connection strategy that adapts to collision information gathered over time. The framework also features scheduling logic between multiple strategies that decreases the failure rate as well as the planning time compared to other competing methods. The scheduling logic regulates the exploitation of focusing hints derived from the workspace obstacles. These new features the reduce average failure rate and improve the average planning time. This allows the point-to-point trajectory planning methods to be used not only in finding collision-free trajectories but also facilitate decision-making at higher levels of the planning stack.
Mobile Manipulator Trajectory Planning
Mobile manipulators are being deployed for transporting parts between machines and work stations in warehouses and shop floors. To increase the efficiency of operations, these mobile manipulators are required to complete the tasks as fast as possible. The task of object transportation has several sub-tasks to it, namely; moving towards the object to be transported, picking-up the object, carrying the object and finally depositing it. Our work focuses on the first three parts of this task, from the point of view of efficient mobile base motion planning, followed by efficient manipulator planning for grasping and finally planning for gripper motion for efficient grasping. Moreover, we also focus on motion planning for carrying large objects with a mobile manipulator in cluttered environments. For motion planning of the mobile base, we use systematic search with motion primitives with specialized heuristics so that the mobile manipulator goes from the starting configuration to goal configuration while picking-up the object in an resolution-optimal manner. This work is followed by motion planning for the manipulator on a moving mobile base in cluttered environments for picking-up objects. Here, we use sampling-based algorithms with specialized techniques to accelerate the motion planning process. Thereafter, in the next work we move the gripper at a speed to grasp objects while moving. The gripper speed needs to be selected to ensure that the pick-up operation does not fail due to uncertainties in part pose estimation. This, in turn, affects the mobile base trajectory. For this we have developed an active learning based approach to construct a meta-model to estimate the probability of successful part pick-up for a given level of uncertainty in the part pose estimate. Using this model, we develop an optimization-based framework to generate time-optimal trajectories that satisfy the given level of success probability threshold for picking-up the part.
Robot and Part Placement Planning
The workspace of a serial manipulator is limited. Additionally, complexities exist due to the joint limit and self-collision-avoidance constraints. The robot workspace includes singularities, and it is generally preferred for a robot to carry out tasks away from singularities. Different locations in the robot workspace impose different constraints on achievable forces and velocities. Variations in achievable forces and velocities can be quite large. Moreover, there might not exist a continuous path between two points in the robot workspace that are far apart. Workspace characteristics can be quite complex for redundant manipulators, and unfortunately, there is no simple model to capture these. Many complex manufacturing tasks such as welding, robotic finishing, composite layup, painting, 3D printing, etc. use manipulators to follow continuous paths under constraints on the workpiece. It is important to find the right location and orientation for the workpiece with respect to the robot to meet all the task constraints. This problem is called a workpiece placement problem. We formulate the problem of identifying a feasible placement as a non-linear optimization problem over the constraint violation functions. This is a computationally challenging problem. Our approach includes successively searching for the solution by incrementally applying different constraints.
Execution Monitoring, Learning, and Plan Refinement
Context-dependent Compensation Scheme to Reduce Robot Trajectory Execution Errors
A robot's repeatability measures how precisely the robot can return to a previously visited (or taught) workspace configuration (e.g., end-effector position and orientation) under the same loading configuration. The accuracy of a robot measures how accurately a robot can reach a specified workspace configuration. This requires inverse kinematics calculations to compute the joint angles and using a feedback controller to achieve the desired joint values. The kinematic errors (e.g., link dimensions, parallelism, orthogonality errors, etc.), joint errors (e.g., sensor errors, etc.), and the non-kinematic errors (e.g., link stiffness, gear backlash, etc.) of the robot result in deviation from the desired end-effector configuration. Currently, automatically generated trajectories cannot be directly used on tasks that require high execution accuracies due to errors accused by inaccuracies in the robot model, actuator errors, and controller limitations. These trajectories often need manual refinement. This is not economically viable on low production volume applications. Unfortunately, execution errors are dependent on the nature of the trajectory and end-effector loads, and therefore devising a general-purpose automated compensation scheme for reducing trajectory errors is not possible. To mitigate these errors, we are developing a method for analyzing the given trajectory, executing an exploratory physical run for a small portion of the given trajectory, and learning a compensation scheme based on the measured data. The learned compensation scheme is context-dependent and can be used to reduce the execution error.
Self-Supervised Learning of Process Parameters
Robots are traditionally used in assembly, machine tending, and material handling tasks. There is increasing interest in using robots for processing applications. In these applications, the robot is required to make changes to the part being processed. Robotic surface finishing tasks such as cleaning, polishing, sanding, machining, and painting are examples of processing applications. Physics-based models are not known a priori for many processing applications. This is often the case when a new material, part, or tool is under consideration. This problem also arises when there is significant uncertainty in the state of the workpiece or the tool. A large number of physical experiments need to be conducted to build a complete physics-based model. This can be done for large volume production. Building a complete model first and then using it for parameter optimization is not practical for low volume production. Moreover, the complete model might not be reused due to the nonrepetitive nature of tasks in small volume manufacturing. Our interest is in developing a self-supervised learning method that can identify the optimal operation and trajectory parameters with a small number of experiments. We are developing a method that optimizes the task objective and meets the task performance constraints. The algorithm makes decisions based on the uncertainty in the surrogate models of task performances. The method intelligently samples the parameter space to select a point for evaluation from the sampled set by determining its probability to be optimum within the set. The iterative method rapidly converges to the optimal point with a small number of experiments. We have validated our approach through physical experiments of robotic scrubbing and sanding.
Tool Path Planning for Ensuring Correct Tool Part Contacts
Tool-Path planning is the foundation for automating many manufacturing processes. Robotic manipulators are increasingly being considered to automate tasks that require complex tool motions. Robotic manipulators provide extra degrees of freedom and are more flexible than traditional automation technologies. However, a tool-path needs to be planned and given to the manipulator trajectory generator as input. The traditional tool-path planning considers the tool to have one contact point, known as the Tool Center Point (TCP). This underutilizes the available flexibility of the manipulator. To make use of the manipulator’s flexibility, multiple contact points or multiple TCP candidates can be considered. These tool contact considerations make the tool path planning problem complex and computationally challenging. We tackle these problems with a novel tool path planning algorithm. Our algorithm incorporates the multiple tool contact points consideration during tool-path planning in an efficient manner to generate a high-quality tool-path in a reasonable amount of run time. The numerous possible tool-paths generated due to the multiple TCP candidates are encoded in a directed graph. The efficient use of collision checking on the tool-path generated by a graph search algorithm is used to converge to the high-quality tool-path quickly.
Alert Generation for Human-Robot Teams
Multi-robot systems are becoming more adept to serve alongside humans in real-world missions as researchers make advancements in fundamental capabilities and resilience. These types of systems will be especially useful for dangerous missions in complex environments, e.g., military operations and disaster response, and reduce the risk to humans. In the context of multi-robot missions, we anticipate that human teammates will serve in a supervisory role to manage resources and make critical decisions because of the gravity of events, including life-or-death situations. At a high-level, human supervisors will ensure that robots are continuously working on tasks that align with the mission objectives, and issue commands that account for the dynamically-changing context. In a mission with considerable uncertainty due to intermittent communications, degraded information flow, and failures, humans need to assess both the current and expected future states, and update task assignments to robots as quickly as possible. We are developing a forward simulation-based alert system that proactively notifies the human supervisor of possible, negatively-impactful events, which provides an opportunity for the human to retask agents to avoid undesirable scenarios. We are also developing methods for speeding up mission simulations and extracting alerts from simulation data in order to enable real-time alert generation suitable for time-critical missions. We have conducted human subject studies and verified our hypothesis that the decision making performance of human supervisors can be improved by introducing forward simulation-based alerts.
Robotic Additive Manufacturing
Additive Manufacturing (AM) is expected to revolutionize manufacturing. The current generation of AM technology has overcome many limitations of traditional manufacturing. However, the current AM technology still needs many improvements. The goal of our research is to realize the next generation AM technologies through use of robotics. Articulated robot arms can significantly expand capabilities of AM processes by allowing material deposition using complex non-planar layers. Many composite parts have thin three-dimensional shell structures. Achieving the right fiber orientation is critical to the functioning of these parts. Printing them using conventional planar-layer AM processes leads to fibers being oriented in the plane of the layer. The capability to deposit the material along non-planar conformal layers can produce parts with the desired material properties. We have demonstrated that using non-planar layers can significantly improve structural performance. We use artificial intelligence-based methods to generate robot trajectories to minimizing the build time. Furthermore, we have been working on conformal multi-resolution 3D printing. Robots can be used to perform multi-resolution printing that finds the best trade-off between build speed and surface finish. We are developing the three-nozzle extrusion system and implemented path planning algorithms using non-planar layers with different resolutions. Finally, AM is not expected to produce high-quality electronics (e.g., processor, sensors) in the near foreseeable future. Our research also focuses on the use of robots for the insertion of externally fabricated components such as sensors, actuators, and energy harvesting components during the AM process.
Machining Tending with Semi-Autonomous Robots
Mobile manipulators can be used for machine tending and material handling tasks in small volume manufacturing applications. These applications usually have semi-structured work environments. Fixed automation with conveyor belts, AGVs, or large industrial manipulators, currently used in large volume manufacturing, is not feasible in these applications because of the expenses and inflexibility. Mobile manipulators offer the flexibility to perform a variety of tasks using a limited number of robots around the facility. However, using a fully autonomous mobile manipulator for such applications can be risky, as an inaccurate model of the workspace may result in damage to expensive equipment. On the other hand, the use of a fully teleoperated mobile manipulator may require a significant amount of operator time. We believe that a hybrid operation mode will be beneficial, in which teleoperation and autonomous motions are combined. For this, we have developed a semi-autonomous, mobile manipulation system, called ADAMMS (Agile Dexterous Autonomous Mobile Manipulation System). It is equipped with cameras and other sensors, and is human-supervised to be used in machine tending operations. The human operator can remain at a remote location, use a user interface to provide a set of high-level instructions for each individual task. The robot autonomously plans motions for executing these tasks and shows a simulation of plan execution to the operator, who can then choose to execute or discard the motion plans. The remote operator can also monitor the motions being executed and stop if a collision is likely to occur. Furthermore, if the autonomous operation is infeasible, the operator can take complete control of the robot and still complete the task in teleoperation mode. As automation in manufacturing applications continues to increase, human operator support of the equipment is expected to become increasingly remote due to spatial, logistical, and safety constraints. We are developing a system to enable a single operator to support multiple platforms distributed throughout a facility through a semi-autonomous operation.
Robotic Surface Finishing
Surface finishing is an important manufacturing process. In high mix applications most of the finishing tasks are non-repetitive in nature and have to be performed manually. These finishing operations for parts with complex geometries can be quite labor intensive, and often ergonomically challenging tasks. Automating the finishing operation can reduce the potential for risk to human health. Automated finishing processes can provide consistent quality and reduce the risk of part damage caused by human error. Finishing is often a time-consuming operation. Automated finishing can increase productivity by reducing the touch labor. Using traditional programming techniques to program robots for complex low-volume parts is time consuming and defeats the purpose of automation. We have developed a collaborative finishing system where human operators work on high level decision making, and the robot assistants carry out the labor-intensive low-level finishing tasks. Our system takes high level task commands from human operators and transforms them into low level instructions for the robot. The inputs to our system are a CAD model, task specifications, desired finishing performance constraints, and process parameter range. We are developing AI-based planning algorithms that automatically convert the high-level tasks into robot motions required for finishing. These algorithms can also help users in figuring out the part placement in the cell for performing finishing operations on large parts. We are also developing algorithms for registering the part in the work cell and refining process parameters based on self-supervised learning.
Robotic Composite Layup
Hand layup is a commonly used process for making composite structures from several plies of carbon-fiber prepreg. The process involves multiple human operators manipulating and conforming layers of prepreg to a mold. The manual layup process is ergonomically challenging, tedious, and limits throughput. Moreover, different operators may perform the process differently and hence introduce inconsistency. We are developing a smart robotic cell to automate the prepreg sheet layup process. The cell uses multiple robots to manipulate and drape sheets over a tool. A human expert provides a sequence to conform the ply and types of end-effectors to be used as input to the system. The system automatically generates trajectories for the robots that can achieve the specified layup. Planning algorithms are capable of (a) generating plans to grasp and manipulate the ply and (b) generating feasible robot trajectories. Our system can generate plans in a computationally efficient manner for complex parts. We are also developing an approach for selecting and placing robots in the cell and description of tools and end effectors needed for utilizing the cell. We have demonstrated the automated layup by conducting physical experiments on an industry-inspired mold using the generated plans. Our system can perform sheet layup at speed comparable to human operators.
Human Robot Collaboration on Assembly Operations
Factories of the future are expected to produce increasingly complex products, demonstrate flexibility by rapidly accommodating changes in products or volumes, and remain cost competitive by controlling capital and operational costs. Humans and robots share complementary strengths in performing assembly tasks. Humans offer the capabilities of versatility, dexterity, performing in-process inspection, handling contingencies, and recovering from errors. However, they have limitations in terms of consistency, payload size/weight, and operational speed. In contrast, robots can perform tasks at high speeds, while maintaining precision and repeatability, operate for long periods of times, and can handle high payloads. However, currently robots require long programming times and have limited dexterity. We are developing a framework to build assembly cells that support safe and efficient human-robot collaboration during assembly operations. Our approach allows asynchronous collaborations between human and robot. The human retrieves parts and places them in the robot's workspace, while the robot picks up the placed parts and assembles them into the product. We are developing technologies for automated plan generation, system state monitoring, and contingency handling.