In the present article we study the form factors of quantum integrable lattice models solvable by the separation of variables (SoVs) method. It was recently shown that these models admit universal determinant representations for the scalar products of the so-called separate states (a class which includes in particular all the eigenstates of the transfer matrix). These results permit to obtain simple expressions for the matrix elements of local operators (form factors). However, these representations have been obtained up to now only for the completely inhomogeneous versions of the lattice models considered. In this article we give a simple algebraic procedure to rewrite the scalar products (and hence the form factors) for the SoV related models as Izergin or Slavnov type determinants. This new form leads to simple expressions for the form factors in the homogeneous and thermodynamic limits. To make the presentation of our method clear, we have chosen to explain it first for the simple case of the $XXX$ Heisenberg chain with anti-periodic boundary conditions. We would nevertheless like to stress that the approach presented in this article applies as well to a wide range of models solved in the SoV framework.