May I suggest a debate format that could allow for some flexibility?
The first thing I would do is to determine eligibility of the parties. They would be able to send a representative (probably the party leader) if their parties either had at least 5% of the vote in the previous election or 5% of the House of Commons seats either at the start or end of Parliament. The Conservatives, Liberals, NDP, Bloc, and Greens would all be eligible to send a representative.
Each party's representative would be able to make a five minute opening statement.
Each party would take part in a question and answer session from the panel. There would be no debate between the representatives during this time.
Parties that agree to debate each other would be given five minutes to debate. For example, the Conservatives and Liberals may agree to have a two-way debate with each other. Therefore, a debating session would take place. However, Harper may not agree to debate Green Party leader, Elizabeth May. He may or may not agree to debate Duceppe during the English debate. Therefore, no debates would take place if there were no mutual agreement.
At the end, each party would be able to make a five minute closing statement.
If Harper, Duceppe, and Layton do not wish to debate May, they do not have to. It will be up to the voters to decide on their fate. They may either be rewarded or punished for their actions. At the same time, at least May would be able to make opening and closing statements, plus answer questions from the panelists.
I don't know if this solution would work, but it's a start.