There are 3 secret rooms in the swamp: 1 with the shrine and another one the boatman takes you to. From this second one you can access room number 3 once you have the air sandals!
So it sounds like you've been to both rooms the boatman takes you to, now you can try to go back to room 2 when you have the air sandals to enter the third secret room!
