Zhiwei Zhu

    W358, SRI International/Sarnoff,
    201 Washington Road,
    Princeton, NJ, 08540.
    609-734-3045 (Office).
    zzhu at sarnoff dot com
I'm currently a Principal Scientist in the Vision Technology, Information and Computing Sciences at SRI International/Sarnoff in Princeton, NJ. I obtained my Ph.D degree from the Department of Electrical, Computer, and Systems Engineering, Rensselaer Polytechnic Institute in Dec. 2005 and my M.S. degree from Department of Computer Science, University of Nevada, Reno in August 2002. In July 2000, I graduated from the Department of Computer Science, University of Science and Technology of Beijing, China with a B.S. degree.

Working Experiences

  • 2016 - now, Principal Scientist, Vision Technology, Information and Computing Sciences, SRI International, Princeton, NJ


  • 2011 - 2016, Senior Computer Scientist, Vision Technology, Information and Computing Sciences, SRI International, Princeton, NJ


  • 2010 - 2011, Senior Member of Technical Staff, Vision & Robotics Laboratory, Sarnoff Corporation, Princeton, NJ


  • 2006 - 2010, Member of Technical Staff, Vision & Robotics Laboratory, Sarnoff Corporation, Princeton, NJ


  • June 2001 - September 2001, Summer Intern, Honda R & D Americas Inc., Mountain View, CA


    • For more information, here is my CV.

    Research Interests

    Visual Odometry, Landmark Matching, Object Detection and Tracking

  • vision-based 6-DOF tracking systems for the localization and navigation of trainees or robots.


  • Real-time visual landmark matching in the real scene for localization without drift indoors/outdoors.


  • I developed a set of non-invasive vison-based technqiues for the human facial behaviors understanding, a brief introduction about my thesis work can be found here . For details, please download my thesis .


    • Journal Publications

    Yongmian Zhang, Qiang Ji, Zhiwei Zhu, Beifang Yi, "Dynamic Facial Expression Analysis and Synthesis With MPEG-4 Facial Animation Parameters". IEEE Transactions on Circuits Syst. Video Techn. 18(10): 1383-1396, 2008.

    Yan Tong, Yang Wang, Zhiwei Zhu and Qiang Ji, "Robust Facial Feature Tracking under Varying Face Pose and Facial Expression". Pattern Recognition Journal, 2007.

    Zhiwei Zhu and Qiang Ji, "Novel Eye Gaze Tracking Techniques under Natural Head Movements". IEEE Transactions on Biomedical Engineering, 2007.

    Wenhui Liao, Weihong Zhang, Zhiwei Zhu, Qiang Ji and Wayne Gray, "Toward a Decision-Theoretic Framework for Affect Recognition and User Assistance". International Journal of Human-Computer Studies, Volume 64, Number 9, pages 847-873, 2006.

    Zhiwei Zhu, Qiang Ji, "Robust Real-time Eye Detection and Tracking Under Variable Lighting Conditions and Various Face Orientations". Computer Vision and Image Understanding, Pages 124-154, Vol.38, No.1, 2005.

    Zhiwei Zhu, Qiang Ji, "Eye and Gaze Tracking for Interactive Graphic Display". Machine Vision and Applications, Pages 139-148, Vol.15, No.3, July 2004.

    Qiang Ji, Zhiwei Zhu and Peilin Lan, "Real Time Non-intrusive Monitoring and Prediction of Driver Fatigue". IEEE Transactions on Vehicular Technology, Pages 1052-1068, Vol. 53, NO.4,July, 2004.

    Qiang Ji, Zhiwei Zhu, "Non-intrusive Eye and Gaze Tracking for Natural Human Computer Interaction". MMI-interaktiv Journal, special issue on eye-gaze tracking for human computer interaction, Nr.6, ISSN 1439-7854, March, 2003.


    Conference Publications

    Zhiwei Zhu, Vlad Branzoi, Mikhail Sizintsev, Nicholas Vitovitch, Taragay Oskiper, Supun Samarasekera, Rakesh Kumar, "AR-Weapon: Live Augmented Reality based First-Person Shooting System", 2015 IEEE Winter Conference on Application of Computer Vision, Jan. 5-9, 2015.

    Zhiwei Zhu, Vlad Branzoi, Michael Wolverton, Louise Yarnall, Girish Acharya, Glen Murray, Nicholas Vitovitch, Supun Samarasekera, Rakesh Kumar, "AR-Mentor: Augmented Reality based Mentoring System", IEEE International Symposium on Mixed and Augmented Reality (ISMAR2014), Sep. 10-12, 2014.

    Ziyan Wu, Zhiwei Zhu, Han-pang Chiu, "Virtual Insertion: Robust Bundle Adjustment over Long Video Sequences", IEEE BMVC, Nottingham, UK, 2014.

    Bogdan C. Matei, Nick Vander Valk, Zhiwei Zhu, Hui Cheng, Harpreet S. Sawhney, "Image to LIDAR Matching for Geotagging in Urban Environments", IEEE WACV, 2013.

    Zhiwei Zhu, Han-pang Chiu, Saad Ali, Raia Hadsell, Taragay Oskiper, Supun Samarasekera, Rakesh Kumar, "High-Precision Localization Using Visual Landmarks Fused with Range Data", IEEE Conference on CVPR 2011, June 21-23, 2011. ((1) Augmented Reality demo; (2) Online tracking demo.)

    Taragay Oskiper, Han-pang Chiu, Zhiwei Zhu, Supun Samarasekera, Rakesh Kumar, "Stable Vision-aided Navigation for Large-area Augmented Reality", IEEE Virtual Reality 2011, March 19-23, 2011 (Best Paper Award!!!).

    Taragay Oskiper, Han-pang Chiu, Zhiwei Zhu, Supun Samarasekera, Rakesh Kumar, "Multi-modal Sensor Fusion Algorithm for Ubiquitous Infrastructure-free Localization in Vision-imparied Environments", IEEE Conference on Intelligent Robots and Systems (IROS) 2010.

    Aveek Das, Oleg Naroditsky, Zhiwei Zhu, Supun Samarasekera, Rakesh Kumar, "Robust Visual Path Following for Heterogeneous Mobile Platforms", ICRA 2010: 2431-2437

    Puri, M., Zhiwei Zhu, Qian Yu, Divakaran, A.,Sawhney, H., "Recognition and volume estimation of food intake using a mobile device", IEEE Workshop on Applications of Computer Vision (WACV), 2009.

    Oleg Naroditsky, Zhiwei Zhu, Aveek Das, Supun Samarasekera, Taragay Oskiper, Rakesh Kumar, "VideoTrek: A vision system for a tag-along robot", IEEE International Conference on Computer Vision and Pattern Recognition (CVPR-09), 2009.

    Zhiwei Zhu, Taragay Oskiper, Supun Samarasekera, Rakesh Kumar, Harpreet S. Sawhney, "Real-time global localization with a pre-built visual landmark database", IEEE International Conference on Computer Vision and Pattern Recognition (CVPR-08), 2008.

    Zhiwei Zhu, Taragay Oskiper, Supun Samarasekera and Rakesh Kumar, "Ten-fold Improvement in Visual Odometry using Landmark Matching", IEEE International Conference on Computer Vision (ICCV-07), Rio de Janeiro, Brazil, Oct. 2007. (Accept rate: 281/1190=23%)

    Taragay Oskiper, Zhiwei Zhu, Supun Samarasekera and Rakesh Kumar, "Visual Odometry System using Multiple Stereo Cameras and Inertial Measurement Unit", IEEE International Conference on Computer Vision and Pattern Recognition (CVPR-07), Minneapolis, MN, June 2007. (Oral, Accept rate: 60/1250=4.8%)

    Zhiwei Zhu, Taragay Oskiper, Supun Samarasekera and Rakesh Kumar, "Precise Visual Navigation using Multi-Stereo Vision and Landmark Matching", SPIE Defense & Security Conference: Unmanned Systems Technology VIII, April 2007.

    Zhiwei Zhu, Taragay Oskiper, Supun Samarasekera and Rakesh Kumar, "An Improved Stereo-based Visual Odometry System", The 2006 Performance Metrics for Intelligent Systems (PerMIS’06), August 2006.

    Wenhui Liao, Yan Tong, Zhiwei Zhu and Qiang Ji, "Robust Object Tracking with a Case-Base Updating Strategy", The 20th International Joint Conference on Artificial Intelligence (IJCAI-06), Hyderabad, India, January 2006. (Accept rate: 258/1353=19%)

    Zhiwei Zhu and Qiang Ji, "Robust Pose Invariant Facial Feature Detection and Tracking in Real-Time", The 18th International Conference on Pattern Recognition (ICPR06), August 2006.

    Zhiwei Zhu, Qiang Ji and kristin P. Bennett, "Nonlinear Eye Gaze Mapping Function Estimation via Support Vector Regression", The 18th International Conference on Pattern Recognition (ICPR06), August 2006.

    Zhiwei Zhu, Wenhui Liao and Qiang Ji, "Robust Visual Tracking Using Case-Based Reasoning with Confidence." The International Conference on Computer Vision and Pattern Recognition (CVPR06), June 2006.

    Zhiwei Zhu and Qiang Ji, "Robust Real-Time Face Pose and Facial Expression Recovery", The International Conference on Computer Vision and Pattern Recognition (CVPR06), June 2006.

    Wenhui Liao, Weihong Zhang, Zhiwei Zhu, and Qiang Ji, "A Decision Theoretic Model for Stress Recognition and User Assistance", The Twentieth National Conference on Artificial Intelligence (AAAI-05) (presentation), 2005.

    Wenhui Liao, Weihong Zhang, Zhiwei Zhu, and Qiang Ji, "A Real-Time Human Stress Monitoring System Using Dynamic Bayesian Network", IEEE Workshop on Vision for Human-Computer Interaction (V4HCI), in conjunction with IEEE International Conference on Computer Vision and Pattern Recognition (CVPR'05), 2005.

    Markus Guhe, Wenhui Liao, Zhiwei Zhu, Qiang Ji, & Wayne D. Gray , "Non-intrusive measurement of workload in real-time", Human Factors and Ergonomics Society 49th Annual Meeting, 2005.

    Zhiwei Zhu, Qiang Ji, "A head motion free gaze tracker with one-time calibration", International Conference on Computer Vision and Pattern Recognition (CVPR05), San Diego, CA, June 2005.

    Xiaozhou Wei, Zhiwei Zhu, Lijun Yin, and Qiang Ji, "Avatar mediated face tracking and lip reading for human computer interaction", Proceedings of ACM Multimedia 2004 (SIGMM), New York, NY, Oct., 2004.

    Zhiwei Zhu and Qiang Ji, "Real Time and Non-Intrusive Driver Fatigue Monitoring", 7th International Conference on Intelligent Transportation Systems, Washington, D. C., Oct., 2004.

    Xiaozhou Wei, Zhiwei Zhu, Lijun Yin, and Qiang Ji, "Face Animation by real time feature tracking", ACM SIGGRAPH 2004 (Posters program), Los Angeles, CA, August, 2004.

    Xiaozhou Wei, Zhiwei Zhu, Lijun Yin, and Qiang Ji, "A real time face tracking and animation system", First IEEE Workshop on Face Processing in Video, in conjunction with IEEE International Conference on Computer Vision and Pattern Recognition (CVPR'04). (IEEE Computer Society), Washington DC, June, 2004.Facial Expression tracking and animation.

    Zhiwei Zhu and Qiang Ji, "Real time 3D face pose tracking from an uncalibrated camera", first IEEE Workshop on Face Processing in Video, in conjunction with IEEE International Conference on Computer Vision and Pattern Recognition (CVPR'04), Washington DC, June, 2004.

    Zhiwei Zhu, Qiang Ji, "3D Face Pose Tracking from an Uncalibrated Monocular Camera". International Conference on Pattern Recognition, for Oral Presentation, August 23-26, 2004.

    Haisong Gu, Qiang Ji, Zhiwei Zhu, "Active Facial Tracking for Fatigue Detection". IEEE Workshop on Applications of Computer Vision, Orlando, FL, USA, December 3-4, 2002.

    Zhiwei Zhu, Qiang Ji, Kikuo Fujimura and Kuang-chih Lee, "Combining Kalman Filtering and Mean Shift for Real Time Eye Tracking Under Active IR Illumination". International Conference on Pattern Recognition, for Oral Presentation, August 11-15, 2002.

    Qiang Ji, Zhiwei Zhu, "Eye and Gaze Tracking for Interactive Graphic Display". 2nd International Symposium on Smart Graphics, June 11-13, 2002, Hawthorne, NY, USA.

    Zhiwei Zhu, Kikuo Fujimura and Qiang Ji, "Real-Time Eye Detection and Tracking Under Various Light Conditions". In Proceedings of ACM SIGCHI Symposium on Eye Tracking Research and Applications, 25-27 March 2002, New Orleans, LA, USA.


    Patents

    S. Samarasekera, R. Kumar, T. Oskiper, Zhiwei Zhu,Oleg Nardisky "Unified Framework for Precise Vision-aided Navigation". United States Patent: 8174568, 05/08/2012.

    Zhiwei Zhu,Taragay Oskiper, Oleg Nardisky, etc. "Stereo-based Visual Odometry Method and System". United States Patent: 7925049, 04/12/2011.

    Qiang Ji and Zhiwei Zhu, "Calibration-free gaze tracking under natural head movement". United States Patent: 7306337, 12/11/2007.

    Kikuo Fujimura and Zhiwei Zhu, "Real-time eye detection and tracking under various light conditions". United States Patent: 7206435, 04/17/2007.

    Zhiwei Zhu, Yan Zhuang, "the MIS of General Vehicles and Drivers". patent number: 2000SR0940, 12 June 2000, P.R. China.