I could agree with you, but then we'd both be wrong. Sure, his drafting record is all over the map. But he got guys like Seymour, Mankins, Wilfork, Mayo, Solder, Jones, Hightower, and McCourty in the first round. Guys like Gronk, Light, Jimmy G, Collins, Vollmer, Branch, Chung, and Thuney in Rounds 2-3. And in later rounds players like Brady, Edelman, Cannon, Koppen, Samuel, White, Flowers, Hernandez, etc. Yes, BB also has a long trail of bums he drafted that did absolutely nothing in the league (so do a lot of other teams and GM's). No need to list the names, I am well aware.
As a comparison, the Colts have been lauded for having great drafts and selected tons of impact players over the same timeframe as NE has been good. With all those great players that they have taken, they have won 1 SB . . . and that was 14 years ago. Drafting is only one piece of the puzzle.
BB has made masterful trades and acquisitions over the years. He added guys like Moss, Welker, Dillon, Revis, etc. Taken guys that were nobodies in other organizations and made them productive in NE. Found UDFA's and outcasts that became huge contributors. He has gamed the compensatory pick system into the gift that keeps on giving every year. Most importantly, he has managed the salary cap better than anyone else ever has . . . I don't think that is even debatable . . . everyone else is vying for second place in the salary cap management category. No one else has been able to get as much out of a team for what they were paying players. And he's been able to do that for decades.
If people want to say Brady has been playing at a deep discount, go ahead and say it (it's not really true, though). Brady has taken most of his money over the years in either upfront signing bonuses or by converting salary into bonuses to shift cap dollars . . . meaning that his low annual salaries completely exaggerate how much money he made. People would see his salary listed as $13 or $14 million when other QB's were getting $27 or $28 million and say he was playing for 50% of the market. Except NE already paid him $10 million in cash, added incentives, or gave him additional bonus money . . . none of which was listed as salary. In reality, Brady was making $23 or $24 million and taking a slight discount.
There was a different thread where we went through and recalculated what Brady made over the course of his career, and he usually was averaged $3-4 million a year under what he could have been making. That difference doesn't go far in staffing a roster. That probably allowed the Patriots to have a better special teamer or a better backup lineman. But it wasn't anywhere near the flexibility that people have been raving about for years. Yes, Brady helped the team out along the way, but it's not like he was taking WAYYYYYY less money as is the perception.
If you want to say that BB = average drafter, I can get on board with that much more easily than I can BB = average GM.