2Department of Rehabilitation Science and Technology, University of Pittsburgh, Pittsburgh, PA
Keywords: Rehabilitation robotics, Vision-based robot control, Alternative robotic manipulation, Human-robot interaction, Assistive technology
Vision-based autonomous control has been investigated as one of the solutions to accommodate people who cannot effectively use the manual control methods [7-15]. Vision-based control can transfer the loading in positioning and fine manipulation to the autonomous algorithm to reduce the complexity exposed to the user. To implement the vision-based autonomous control, many researchers adopted an eye-in-hand camera [7, 8, 13, 15], on the robot gripper or wrist to guide the robot towards an object of interest. This approach needs to update the object locations continuously until the end-effector acquired the target object, and thus is computationally expensive. Some researchers mounted a camera on the fixed position at the robot base or shoulder [12]. While this approach has advantage of finding a path and grasping plan even when the object is occluded from the starting location or folding position [14], it requires the knowledge of the target object as well as the surroundings in advance to localize the target object and plan a trajectory [16]. Other researchers use the combination of the above two approaches to provide more reliable and robust control [10, 14]. However, the combined approach can significantly increase the implementation cost and system overhead. More recently, the use of 3D camera like Microsoft Kinect has been investigated in assistive robot applications [11].
We have developed and evaluated a vision-based Assistive Robotic Manipulation Assistance Algorithm (AROMA) for JACO, which uses another kind of low-cost 3D depth sensing camera and an improved Inverse Kinematic (IK) algorithm over the IK algorithm provided by the JACO Application Program Interface (API). In addition, AROMA was developed on a Windows operating system instead of the Robot Operating System (ROS), which makes it easier for the algorithm to be adopted by nontechnical users and clinical professionals
Another problem is the JACO workspace and positioning accuracy. According to the JACO technical documents, JACO can reach approximately 90cm in all directions using joystick control. However, when using the default IK function, we noticed that JACO has a decreased working space due to the embedded singularity avoidance algorithms. To examine the actual workspace where there is no limitation in performing translational, rotational and grasping motions, we programmed the JACO to automatically reach and perform all three basic motions (translational, rotational, and grasping) at 1cm resolution within the theoretical workspace of 90cm radius and 110 degrees of phi φ Figure 3. The JACO robot arm was found fully functional within the area 3 (i.e., a quarter-ellipsoid with about 62cm radius and 110 degrees of
To address these issues, we developed a custom IK module that considered the missing tip-target link kinematic based on the target object pose through the 3D vision module and the robot parameters of the JACO robotic arm as shown in Table 1-3. To compute minimum effort IK solution and plan trajectory to reach to a desired goal position, we adopted OpenRAVE's IKFast robot kinematics compiler, which analytically solves and generates optimized IK functions. We also adjusted dynamics caused by common factors like gravity and positioning tolerances by refining the IK solution using the Levenberg-Marquardt algorithm, also known as a damped least square method [18, 19]. The refined IK solution was sent to the JACO controller where virtual joystick signals emulating physical joystick commands were used to control JACO.
The arm sagging issue, that is the hand position of the JACO arm drops down 1-2cm whenever grasping commands are sent, was solved by using Cartesian command information (API function: GetCommandCartesianInfo()), instead of relying on the reported current arm position (API function: GetHandPosition()).
|
alpha(i-1) |
a(i-1) |
di |
theta1 |
1 |
0 |
0 |
D1 |
q1 |
2 |
-pi/2 |
0 |
0 |
q2 |
3 |
0 |
D2 |
0 |
q3 |
4 |
-pi/2 |
0 |
d4b |
q4 |
5 |
2*aa |
0 |
d5b |
q5 |
6 |
2*aa |
0 |
d6b |
q6 |
|
Link length values (meters) |
|
D1 |
Length |
Explanation |
0.2102 |
Base to elbow |
|
D2 |
0.4100 |
Arm |
D3 |
0.2070 |
Front arm |
D4 |
0.0750 |
First wrist |
D5 |
0.0750 |
Second wrist |
D6 |
0.1850 |
Wrist to the hand |
Alternate parameters |
||
aa |
((11.0*PI)/72.0) |
|
ca |
(cos(aa)) |
|
sa |
(sin(aa)) |
|
|
(cos(2*aa)) |
|
|
(sin(2*aa)) |
|
d4b |
(D3 + (ca-c2a/s2a*sa)*D4) |
|
d5b |
(sa/s2a*D4 + (ca-c2a/s2a*sa)*D5) |
|
d6b |
(sa/s2a*D5 + D6) |
As for the success rate, grasping the small ball was least reliable with 93% success rate. In the failed trials, the JACO hand was able to pick up the ball but then dropped it off before reaching to the target height. We speculated that the glossy surface of the small ball might compromise the object pose estimation and thus lead to unreliable grasping points. Table 5 shows the deviations between actual object widths and estimated ones for each ball. The small ball not only had the largest variance among the three balls, but also had tendency to underestimate the object size. The success rate of the bottle grasping test was 85%. The failed trials were mostly due to the collisions between the JACO hand and the bottle. For both the small ball and bottle experiments, the object locations were well distributed and the locations (marked in red) where unsuccessful trials occurred were highly scattered and no systemic pattern was found Figure 6 and Figure 7. In addition to
Ball Size |
Ball Experiment |
Bottle Experiment |
|||
Average Grasping Time(sec) |
Success Rate |
Average Grasping Time(sec) |
Success Rate |
||
|
5.51 (±1.38) |
93/100 |
5.96 (±1.95) |
17/20 |
|
M |
4.17 (±0.97) |
100/100 |
|||
L |
4.46 (±1.48) |
100/100 |
Width (mm) |
Small |
Medium |
Large |
Actual |
45 |
65 |
85 |
Estimate |
39.1(±4.57) |
64.3(±2.57) |
84.86(±2.28) |
Second, the AROMA uses infrared images, and thus is less dependent on ambient lighting conditions than conventional image processing which requires images of an object under different lighting conditions or sources in order to improve the algorithm invariance to diverse lighting conditions. Tsui, et al. developed a vision-based autonomous system for a wheelchairmounted robotic manipulator using two stereo cameras, one mounted over the shoulder on a fixed post and one mounted on the gripper. Once the user only needed to indicate the object of interest by pointing to the object on a touch screen, the autonomous control automatically took over the rest of the task by reaching towards the object, grasping it, and bringing it back to the user [10]. They evaluated this system with 12 individuals with various physical and cognitive disabilities, where participants were asked to retrieve an object from a bookshelf. The success rate of the autonomous function was 65% (129/198). Of the 69 unsuccessful trials, 56 (81%) were due to algorithm failures. Jiang, et al. also developed a vision-based autonomous robot control system combining a JACO robot arm with two Microsoft Kinect sensors: one for recognizing user voice, gesture and body part; the other for object recognition [11]. User's voice and hand gestures were used as the robot control commands. The object recognition algorithm relied on a two-step process, which extracted the feature vector for an object using Histogram of Oriented Gradients algorithm, then trained the model and classified the objects applying nonlinear support vector machine algorithm. The system was evaluated by one participant with four different manipulation tasks (5 trials per each), including, drinking, phone calling, taking a self-portrait, and taking photos of the surroundings. The performance time ranged from 14-130 seconds and accuracy ranged from 52-98%.
Third, AROMA addressed the inherent limitations of the JACO onboard IK algorithm, including the missing tip-target link, reduced working space, and arm sagging issues. In addition, the AROMA was developed under a Windows operating system, making it not only easier to integrate new and existing alternative input devices without developing additional driver software, but also increasing the likelihood of adoption by users and clinical professionals.
However, AROMA has also several limitations. First, when dealing with the missing tip-target link, we aimed to find a goal configuration of the end-effector that matches the target object pose under the assumption that there is no obstacle between the manipulator and the target object. This may compromise the manipulation performance and safety in challenging environments such as cluttered space. To address this issue, additional sensors such as an eye-in-hand camera or force/ tactile sensors could be adopted. Second, the experiments were conducted with simple shaped objects with smooth surfaces. To accommodate a variety of everyday objects with different characteristics, the damping factor for the DLS method may need to be adjusted to achieve a balance between performance stability and speed. Lastly, it is also important to apply AROMA to real-world manipulation tasks and test it with individuals having upper extremity impairments.
In addition to supporting autonomous operation of JACO, an practical application of AROMA is to support semi-autonomous control, where direct user control is combined with robot autonomy, strategically reducing the complexity exposed to the user while keeping the user in the control loop [20]. Users usually find fine manipulation of a robot manipulator more challenging and spend more time on adjusting the end-effector position and orientation before grasping. AROMA could potentially address this issue by allowing users to use conventional input methods (e.g., joystick) to move the arm close to the target object, and then user voice control to command the robot for fine manipulation. e.g., grasping or pushing. Kim, et al. found that while user effort required for operating the robot with autonomous control was significantly less than with the manual control, user satisfaction with the autonomous control was lower than with the manual control [13]. With the semi-autonomous control, users only need to control the gross motion and leave the fine manipulation to AROMA, which could potentially lead to improved performance and satisfaction.
We are planning on two follow-up studies to apply the AROMA. One study is to apply the semi-autonomous approach to an overhead track mounted assistive robotic system called KitchenBot [21], which operates along an overhead track built into the kitchen to assist individuals with physical disabilities for tasks in a typical kitchen environment. Another study is to combine AROMA with automatic speech recognition to provide complete hands-free semi-autonomous operation.
- Allin S, Eckel E, Markham H, Brewer BR. Recent trends in the development and evaluation of assistive robotic manipulation devices. Phys Med Rehabil Clin N Am. 2010;21(1):59-77.
- G. Romer, Stuyt HJ, Peters A. Cost-savings and economic benefits due to the assistive robotic manipulator (ARM), in Rehabilitation Robotics. 2005;201-204.
- Maheu V, Frappier J, Archambault PS, Routhier F. Evaluation of the JACO robotic arm: Clinico-economic study for powered wheelchair users with upper-extremity disabilities. Rehabilitation Robotics. 2011;1-5.
- Römer GW, Stuyt H, Peters G, Woerden KV. 14 Processes for Obtaining a Manus(ARM) Robot within the Netherlands. Advances in Rehabilitation Robotics. 2004;221-230.
- G. Romer, Stuyt H, Peters A. Cost-savings and economic benefits due to the assistive robotic manipulator (ARM). Proceedings of the 2005 IEEE. 9th International Conference on Rehabilitation Robotics. 2005;201-204.
- King CH, Chen TL, Z. Fan Z, Glass JD, Kemp CC. Dusty: an assistive mobile manipulator that retrieves dropped objects for people with motor impairments. Disabil Rehabil Assist Technol. 2012;7(2):168-179.
- Driessen B, Kate TT, Liefhebber F, Versluis AH, Woerden J. Collaborative control of the manus manipulator. Universal Access in the Information Society. 2005;4(2):165-173.
- Tijsma H, Liefhebber F, Herder J. Evaluation of new user interface features for the manus robot arm, in Rehabilitation Robotics, 2005. ICORR 2005. 9th International Conference. 2005;258-263.
- Laffont l, Biard N, Chalubert G, Delahoche L, Marhic B, Boyer FC, et al. Evaluation of a graphic interface to control a robotic grasping arm: a multicenter study. Arch Phys Med Rehabil. 2009;90(10):1740-1748.
- Tsui KM, Kim DJ, Behal A, Kontak D, Yanco HA. I want that: Human-in-the-loop control of a wheelchair-mounted robotic arm. Applied Bionics and Biomechanics. 2011;8(1):127-147.
- Jiang H, Zhang T, Wachs JP. Autonomous Performance of Multistep Activities with a Wheelchair Mounted Robotic Manipulator Using Body Dependent Positioning. 2014.
- Chung CS, Wang H, Cooper RA. Autonomous function of wheelchair-mounted robotic manipulators to perform daily activities. IEEE Int Conf Rehabil Robot. 2013;1-6.
- Kim DJ, Lovelett R, Behal A. Eye-in-hand stereo visual servoing of an assistive robot arm in unstructured environments. ICRA'09. IEEE International Conference. 2009;2326-2331.
- Srinivasa SS, Ferguson D, Helfrich CJ, Berenson D, Collet A, Diankov R, et al. HERB: a home exploring robotic butler, Auton Robot. 2010;28:5-20.
- Tanaka H, Sumi Y, Matsumoto Y. Assistive robotic arm autonomously bringing a cup to the mouth by face recognition. Advanced Robotics and its Social Impacts (ARSO), 2010 IEEE Workshop. 2010;34-39.
- Corke PI. Visual Control of Robots: high-performance visual servoing: Research Studies Press Taunton. 1996.
- Modarress D, Svitek P, Modarress K, Wilson D, Micro-Optical Sensors for Boundary Layer Flow Studies, in ASME 2006 2nd Joint US-European Fluids Engineering Summer Meeting Collocated With the 14th International Conference on Nuclear Engineering. 2006;2:1037-1044.
- Wampler CW. Manipulator inverse kinematic solutions based on vector formulations and damped least-squares methods. Systems, Man and Cybernetics. IEEE Transactions. 1986;16(1):93-101.
- Nakamura Y, Hanafusa H. Inverse kinematic solutions with singularity robustness for robot manipulator control, Journal of dynamic systems, measurement, and control. 1986;108(3):163-171.
- Kemp CC, Edsinger A, Torres J, Challenges for robot manipulation in human environments, IEEE Robotics and Automation Magazine. 2007;14:20.
- Ding D, Hyun K, Cooper R, Telson J, Kavita K, Focus Group Evaluation of an Overhead Kitchen Robot Appliance.RESNA, Indianapolis. 2014.