This article proposes a vision-based target motion analysis and collision avoidance method for unmanned surface vehicles. To estimate the relative position of the target, the calibrated camera model and camera height constraint are used. In addition, the optical flow equation is adopted to estimate the relative velocity of the target. The estimated relative position and velocity are then integrated using the Kalman filter in order to reduce the effect of measurement noise. Once the kinematic information of the target is determined by the vision-based target motion analysis, the collision risk is calculated by a fuzzy estimator. Based on the collision risk, vision-based collision avoidance control is performed. To validate the effectiveness of the suggested method in various encounter situations, a collision avoidance simulation is conducted.