Approaching Hollesley (pronounced Hoze-ley) from north or west, you come out of the forests to find the heaths and marshes of the Bawdsey Peninsula spread out before you. Behind are memories of a darker time, for during the Cold War there were two massive American airbases in this forest, dictating the line of roads and, mercifully, the abundance of trees. Today, Woodbridge and Rendlesham bases are being developed for residential and commercial use, although parts are still cordoned off, waiting for military installations to be either removed or to find a new role. Beyond the village, the sea appeared for the first time that day, and the fortress between me and it was Hollesley Bay Prison Colony, erstwhile home of Hollesley's most famous former resident, the playwright Brendan Behan, and setting of much of his screamingly funny autobiography, Borstal Boy. Beyond was a fortress of an earlier age, one of the Martello towers that line the Suffolk coast from Aldeburgh southwards. It is set at the end of Shingle Street, a strange, remote community, site of a famous hushed-up incident in World War II. Supposedly, there was an accident, and hundreds of people died.

But every county has a story like this; the only real controversy, according to a local I spoke to, was that the RAF had used the pub for target practice. In any case, Shingle Street is more memorable to me for the wild party on the beach there the day after I got married in 1990. After such excitement, Hollesley is a remarkably suburban place, and the biggest place on the peninsula, with a proper shop, a school, and a pub. Presumably many of the local people work at the nearby prison.

All Saints is set in the middle of the village, just off the High Street, up a little rise. A house sits next to it in a cosy juxtaposition, its south wall decked in wisteria. All Saints looks terribly polite in its churchyard, and the Victorians gave it a good seeing to. Curiously, when they knocked the wall through to build a north aisle, they found a medieval arcade hidden in the stonework - obviously, this church had been made smaller after the Reformation in the same way as nearby Bawdsey. The tower is mid-15th century, with a little niche above the west door. Simon Cotton tells me that there were bequests for the tower in 1452 (of about £12) and 1465. A bequest was made for a bell in 1495. There are Marian monograms on the battlements.

Ten years ago, when I last cycled around this part of Suffolk, this was the only church on the Peninsula which I found closed to me. Coming back this way, I asked a helpful chap at nearby Shottisham where I might find the Hollesley key. "Don't worry about that", he laughed. "The key is under the bench in the porch!" But he was having a joke, because Hollesley church has no porch, and the church is now open to pilgrims and strangers every day.

You step into a clean, bright, well-kept interior, filled with the beautiful light of a jewel-like window set in the east end of the north aisle. This depicts the Holy Family, and is the excellent work of the Welsh artist Meg Lawrence, who we have already met at Ormesby St Margaret.

There is a fine royal arms for Charles II, but the other memorable feature of All Saints is the set of unusual bench ends. As at several Suffolk churches, they were the work of an early 20th Century Rector here, and include such curiosities as St Agatha bearing her breast, a three-dimensional rendering of the sciapod, a mythical beast found in medieval form at Dennington, and a sphinx which recalls the Tutankhamun-craze of the 1920s.

Going around to the south door, There is something most curious. Set in the eastern buttress of the tower is a huge holy water stoup, quite out of scale with the door. It is the sort of thing that would have given the iconoclasts a fit of apoplexy, but would no doubt have had the Victorians rubbing their hands in delight. The more you look at it, the more curious it seems. It isn't just the lip that is uneven, the whole base seems to have been set at an angle. The coving has two most curious points cut into it, that look like the tops of the recesses of flushwork. And that is almost certainly what they are, for what we have here is a Victorian confection, a faux-stoup. Cautley thought that the bowl might even have been a mortar, and the alcove has been hacked out of the core most crudely. It pleased the Oxford Movement restorers to make the church seem even more medieval than it already was, even to the extent of suggesting to us that medieval Holleslians were all left-handed - unfortunately, they placed it on the wrong side of the door.