What agreement do you have with the agency as opposed to the landlord?
Any idea what agreement the landlord have with the agency?
Ultimately, your tenancy agreement is with the landlord, unless you have already agreed otherwise with in terms of possible agency fees for early termination. What of that £400+ are demanded by the landlord on you, what of it is just the agent wanting it?
(3) A payment is a permitted payment if it is a payment to a letting agent in consideration of arranging the termination of a tenancy at the tenant's request—
...
(4) But if the amount of the payment exceeds the reasonable costs of the letting agent in respect of the termination of the tenancy, the amount of the excess is a prohibited payment
Go ahead and argue what you think is reasonable costs, but you have to remember your landlord don't have to agree on your early termination. They can always choose the option of you remaining a tenant and paying the final 4 months rent.