You can change the target mode count by entering Modal properties in Analysis/Modal analysis case from the structure tree. The repetition of the error indicates that the number is not sufficient and the number of modes is increased again. In addition, by choosing Ritz vectors from the modal calculation methods, you can provide as much mass participation as required with less mode number.
...