Helicopter unmanned aerial vehicles (UAVs) can be extensively used for military missions as well as in civil operations, ranging from multi-role combat support and search and rescue, to border surveillance and forest fire monitoring. Helicopter UAVs are underactuated nonlinear mechanical systems with correspondingly challenging controller designs. This paper presents an optimal controller design for tracking of an underactuated helicopter using an adaptive critic neural network (NN) framework. The online approximator-based controller learns the infinite-horizon continuous-time Hamilton–Jacobi–Bellman (HJB) equation and then calculates the corresponding optimal control input that minimizes the HJB equation forward-in-time without using value and policy iterations. In the proposed technique, optimal tracking is accomplished by a single NN, which is tuned online using a novel weight update law. Stability analysis is performed and simulation results demonstrate the proposed control design.