The problem is that Singer has already agreed to direct fairy tale Jack the Giant Killer for Warner Brothers. He can't drop out at this point as location scouting has already begun, and Fox is loathe to wait until 2012 for Singer to be freed up again to work on the next X-men film. Fox has already begun actively seeking a replacement, and by all accounts the names on their list are directors who have a history working with comic book properties and will make fanboys happy.
That sounds like Zack Snyder and Matthew Vaughn, to me. Snyder obviously has serious superhero cred after helming both 300
It's possible that Singer may not be able to make his return to the X-men franchise for at least a couple of years, maybe just in time for X-men 4.