Plans for the estate in Victoria Road are set to go before Bolton Council's planning department next week but at a meeting of Horwich Town Council, councillors voted to recommend the application is thrown out.
Developer Bowsall Ltd has applied for permission to redevelop the site, which will be vacated in the summer, with the college relocating to new town centre facilities.
One of the main areas of contention revolves around developers intention to build on the playing field next to the college building.
Cllr Richard Silvester said: "That playing field should not be developed. It should be saved completely.
“The main thing is I think the development should be reduced in size. Unfortunately the college building is on brownfield land and can be developed on, but I think the playing field should be left out, reducing the number of houses from 140 to 100.”
He also called for a new 20mph zone for the area as well as new traffic signs.
The council voted to recommend refusal on the plans but to add that if approval is granted, Cllr Silvester’s recommendations are taken on board.
The meeting on Thursday also heard from John Shannon, chair of the governing body of St Mary’s RC Primary School, which is next to the proposed development site.
At a previous meeting he raised concerns at the impact the housing estate could have on the primary school’s need to expand in the future.
He told councillors: “At the last meeting, the council deferred a decision because I told you I was waiting to hear back from the developers regarding our concerns.
“I just wanted to let you know that since then I have had no contact from Bowsall.”
Cllr Kevan Helsby described the plans as “going in completely the wrong direction”.
He added: “This is like planning gone mad. We are moving away from education and sport and towards more housing.”