For what it's worth it takes longer at certain places if there is no winning tradition to be rebuilding to. Brown doesn't exactly have a great, storied tradition of winning in the last 10-12 years. There were a couple of years in there where they were solid and finished high in the conference, but they failed to build on that and bottomed back out. For all intents and purposes, Brendan Whittet is trying to build something worse than an expansion program; he's trying to convince us that the Cleveland Browns can become the New England Patriots.
FWIW, that earns him at least a 6-7 year reprieve as opposed to 2-3 year turnarounds. He needs to get his guys in here just to tread some form of water before he can land recruits. Recruiting hasn't been the issue, evidenced by the fact that Brown got Roy before Northeastern poached him. He's had some solid recruits. He just needs to put it all together. I'd give him an additional 2-3 years on top of what he's done already. If he can't turn it around in the next couple of years, then Brown will need to make a change.
Programs like Brown need 7 years to rebuild. It takes a coach 4 just to implement his system and get a full complement of players just to demo the system. He needs to go from demonstrating a system with guys who aren't his to demonstrating a system with lower-quality guys who he brought in to bringing in the big guns. That's a huge process.