'It was heartbreaking to attend a fertility appointment in a maternity unit'