Council investigating moving 3 practices into the old magistrates building.
The proposals affecting thousands of patients have been put forward by the Brunel Medical Practice which has its main site at Babbacombe and branches in St Marychurch and Shiphay.
Patients have been told the move is needed because the current premises are not suitable to provide modern health services.
Concerns have been raised about the move out of communities and the extra travel for patients if the three outlying surgeries are closed.
The issue was raised by St Marychurch councillor Hazel Foster at a meeting of Torbay Council’s overview and scrutiny board.
Councillors decided to set up a group to examine the proposals in more detail.
The practice was rated Good overall by the Care Quality Commission after an inspection in 2018. At the time it had 16,000 patients registered.
The council’s deputy leader Darren Cowell, an Independent who represents Shiphay, said there was “a lot of anxiety” among patients.
He said the court building was not well served by bus routes and the move went against the desire to provide services close to communities.
He acknowledged the pressures facing the practice, which is finding it difficult to recruit staff and provide a full range of services, and the benefits to the town centre economy by bringing in more people.
Cllr Cowell said: “I understand where they are coming from, but I’m still very concerned about the loss of that community facility.”
The court building in Union Street, Torquay, has been empty since it was closed in 2017.
A letter has been sent to patients setting out the reasons for the merger plan and asking for feedback.
The letter from practice manager Rachael Lankshear says: “The Partners believe that in order to be able to offer excellent care to our patients now and in ten years from now we must move to one larger, modern health centre.
“We have identified the former Magistrates Court at Castle Circus in the Town Centre as our preferred site and have had an offer accepted to purchase it.
“We believe that working together from one site would allow us offer more appointments, have extra staff to answer the phones and would allow us to offer additional services to our patients.
“We currently have no free clinical rooms and are struggling to recruit new staff to our team.
“We feel that a new health centre at Castle Circus would encourage people in to the town centre, which would also be good for our local economy.
“Our plans would not prevent patients from seeing their usual doctor or result in less appointments being made available.
“We believe that relocating in this way would safeguard the future of our practice and would benefit all patients.”
Senior managers from the practice were due to be at a public meeting to discuss the plan.
The meeting was being hold on Tuesday, February 25, at 7pm at the St Anne’s Hall Community Centre, in Babbacombe Road.