Yes, and a steady stream of people are doing exactly that. The complication is that the area carries 186 listed buildings and five conservation areas covering the town and the villages around it. Inside those lines, the things you would normally do without asking become things you ask about first. Outside them, this is an ordinary Somerset town where a rear extension is a rear extension. Which side of the line your house sits on decides how the next two years of your life go.
Of those 186 listed buildings, three are Grade I, 13 are Grade II star, and 170 are Grade II. Listing is not a rule about the front elevation. It applies to the building as a whole, and it means listed building consent is needed for alterations that would otherwise sail through under permitted development. Owners of Grade II properties sometimes assume the lowest grade means the lightest touch. It means the same consent process with a slightly shorter conversation.
The five conservation areas cover Castle Cary itself along with Ansford, Pitcombe, Hadspen and Higher Flax Mills. Conservation area status works differently from listing: it protects the character of a place rather than a specific building, which in practice restricts some permitted development rights, brings demolition under control, and puts trees under notice requirements. A house that is not listed but sits inside one of these areas can still find that windows, cladding, roof coverings and anything visible from the street are matters for the planning department rather than for you and a builder.
Five scheduled monuments sit within the area as well. That is the strictest protection tier in the system, and for most buyers it is irrelevant until it suddenly is not, which happens when a plot adjoins one. If a property borders open ground with an archaeological designation, works to the land are a different and slower conversation than works to the house.
None of this stops people building. Somerset's planning records carry 164 applications across the area between April 2024 and June 2026, with 82 of those in the last twelve months. Whatever the constraints look like written down, there is an active flow of householders extending, converting and altering, which is the most useful thing to know before assuming the town is closed for business.
Two listed buildings in the area sit on Historic England's Heritage at Risk register. That is a statement about the condition of those specific buildings and the scrutiny and potential grant support attached to their repair. It is not a warning about the ordinary house down the road, and it should not read as one.
Practically, the sequence matters more than the ambition. Establish whether the address is listed and whether it falls inside a conservation area before you make an offer, not after the survey, because both facts are public and both change what the property is worth to you. Where a house is unconstrained, standard permitted development applies and the project is as boring as it is anywhere else. Where it is constrained, budget for a pre-application conversation with the planning department and for a design that starts from what the building will tolerate rather than from what the extension brochure suggested.
Freedom to alter a home is the weakest thing this town offers, and it is not close. The compensation is that the constraints are the reason Castle Cary looks the way it does, which is presumably why you were looking at it in the first place.