The Colyseum (yeah, that is how it's spelled in this game) is north of Fort Zenna and slightly west. You go up between 2 lines of trees.
It's not just a question of levels; good gear makes a big difference as well, so finding things like the Slaughterer Gauntlet is a great help.