We consider Abelian-by-cyclic groups for which the cyclic factor acts by hyperbolic automorphisms on the Abelian subgroup. We show that if such a group acts faithfully by $C^1$ diffeomorphisms of the closed interval with no global fixed point at the interior, then the action is topologically conjugate to that of an affine group. Moreover, in case of non-Abelian image, we show a rigidity result concerning the multipliers of the homotheties, despite the fact that the conjugacy is not necessarily smooth. Some consequences for non-solvable groups are proposed. In particular, we give new proofs/examples yielding the existence of finitely-generated, locally-indicable groups with no faithful action by $C^1$ diffeomorphisms of the interval.