This paper proposes an effective novel cuckoo search algorithm (ENCSA) in order to enhance the operation capacity of hydrothermal power systems, considering the constraints in the transmission network, and especially to overcome optimal power flow (OPF) problems. This proposed algorithm is developed on the basis of the conventional cuckoo search algorithm (CSA) by two modified techniques: the first is the self-adaptive technique for generating the second new solutions via discovery of alien eggs, and the second is the high-quality solutions based on a selection technique to keep the best solutions among all new and old solutions. These techniques are able to expand the search zone to overcome the local optimum trap and are able to improve the optimal solution quality and convergence speed as well. Therefore, the proposed method has significant impacts on the searching performances. The efficacy of the proposed method is investigated and verified using IEEE 30 and 118 buses systems via numerical simulation. The obtained results are compared with the conventional cuckoo search algorithm (CCSA) and the modified cuckoo search algorithm (MCSA). As a result, the proposed method can overcome the OPF of hydrothermal power systems better than the conventional ones in terms of the optimal solution quality, convergence speed, and high success rate