Some of what you are asking are legal questions that a lawyer, admitted to practice in your jurisdiction, would have to answer.
But check the fine print of the contract. You do have a copy of it, yes? And find out if there was a covenant not to compete that you signed (they aren't legal everywhere) or if there are any provisions for them hitting you with special damages. And see if there is a paragraph (there should be) regarding where jurisdiction would be in the event of a suit against you for breach. This paragraph would be in their (choice of jurisdiction or venue) because it's an entirely possible scenario that you would live in one state, the agency would be headquartered in another, and the job they sent you to would be in a third. Such paragraphs do not decide things but they can sway a court if you are sued. And, yeah, there is the possibility that you would be sued - there would not be a contract if the agency wasn't trying to protect their legal rights (but keep in mind that there are no guarantees that you would or would not be sued - but it is a possibility). Fortunately you, too, have rights.