Oracle Temple

I am in this place and feel like I am getting nowhere, I have looked in all directions and come to nothing but dead ends and in the south path I have found a small pink castle and a little Japanese house looking thing and whilst I have tried rearranging my party (placing each of the characters in the leader position) I can't seem to do anything now matter how many times I try interacting with both structures (and statues and whatever else I come across.) What am I do wrong? I can't do anything else as all ways are dead ends, am I missing something?