The Island is home to a ton of great annual and totally random one off festivals and events.  This list is by no means exhaustive but it gives you a place to start and an idea as to what is within reach here from cultural and historical events to world class sporting events, food and wine festivals and the list rolls on……
Festival of Trees Nov – Jan
Habitat for Humanity 9th Annual Gingerbread Showcase Nov- Jan
Wildlife Photographer of the Year Exhibition at Royal BC Museum Dec – Apr
Victoria Film Festival   Feb-Mar
Song & Surf Music Festival Feb
Capital City Comic Con Mar
Greater Victoria Performing Arts Festival    Apr-May
Uno Festival -Contemporary solo theatre and performance   May
Victoria Highland Games and Celtic Festival    May
Swiftsure International Yacht Race    May
TD Victoria International Jazz Festival    June -July
Victoria Ska Fest    June
Victoria Pride    July
Phillips Backyard Weekender Music festival   July
Vancouver Island Music Festival (Comox Valley)   July
Rock the Shores Music Festival   July

Free B Film Festival Aug
Symphony Splash Aug

Victoria Dragon Boat Festival    Aug
Victoria Fringe Theatre Festival    Aug-Sept
Victoria Buskers Festival Aug-Sept
Vancouver Island Blues Bash Sept
Victoria International Cycling Festival
Victoria Classic Boat Festival    Sept
Rifflandia Music Festival    Sept
Goodlife Fitness Victoria Marathon    Oct
Art of the Cocktail    Oct

Other great local resources for event listings include:
Tourism Victoria Calendar of Events

Downtown Victoria Business Association (DVBA) Calendar of Events

Tourism Vancouver Island Festivals and Events

Chatterblock Victoria – especially great for families with young kids