Well I gave it a sand down tonight. I know it's nowhere near finished but it's getting hard to see where the filler is required, so I've given the duct a scratch coat. Most of the imperfections is where different filler was used, and I would imagine would get covered in a few coats of paint. I can now work on filling the effected areas:
