For almost everyone in Sway, the answer is yes. The village carries just four listed buildings in total, and every one of them sits at the lowest grade of protection. There are no conservation areas recorded across the place and no scheduled monuments in the built-up area itself. Unless you have bought one of a tiny handful of protected homes, the heritage rulebook has very little to say about what you do to your walls.
Four listed buildings is a thin web of protection for a whole village, and the two you are most likely to have heard of are landmarks rather than houses. Peterson's Tower, the concrete folly that gives Sway its skyline, is listed, as is the Church of Saint Nicholas. Both sit at grade two, the category that covers the overwhelming majority of listed structures in the country and the one that comes with the lightest touch. Listing at that level does not freeze a building in aspic. It means alterations that affect its character need consent first, and that a planning officer gets a say in what you do to the fabric. What it does not do is spread that burden across the street. The protection attaches to the specific building, so your neighbours in an unlisted house three doors down are working under an entirely different, and far freer, set of rules than the owner of the tower ever will.
In practice that splits Sway into two populations. If your home is not on the list and not inside a conservation area, and almost no home here is either, then the ordinary permitted development freedoms apply and a good chunk of what people want to do, a rear extension, a loft conversion, a bit more space at the back, sits within them or clears a routine application. The one piece of context worth respecting is that Sway sits inside the New Forest national park, which is the backdrop every application here is read against, so the sensible move is a quick check with the local planning team before the builders arrive rather than an assumption that anything goes. That is a phone call, not a fight. For the owner of one of the four listed buildings the calculus is different, and consent for changes to the fabric becomes the first job rather than an afterthought. For everyone else, the paperwork is closer to a formality than an obstacle.
Sway is one of the freer places you could pick if extending or altering a home is on your mind. With only four listed buildings, no conservation areas and no monuments dropped through the middle of the village, the heritage constraints that tie owners in knots elsewhere barely register here. Buy an ordinary house and the main limit on what you build is your budget and your builder, not a filing cabinet at the council.