This is a problem that falls outside of the protocol I think, but whatever solution that is proposed can probably be solved on contract level. I can imagine a solution where a contract collects multiple oracle responses and decides what the outcome will be.
Withholding fees, or distributing them between oracles on the protocol level is a bit more problematic, and it would have a long feedback loop. Any problems would need a hard fork to be solved, while a problem on the contract level can be solved by posting a new contract.
Personally, I prefer keeping the protocol as simple as possible, so that you have a chance of reasoning, and even proving stuff about it. For oracles, the protocol is not trivial, but it is relatively simple. If the oracle responds with an answer that fulfills the requirements of TTL and ABI, it has done its work and gets the reward. Reasoning about the qualitative value of the answer would complicate matters. And having the original question poster accept the answer would open the door to malicious behavior as well.
I think the best way of looking at the general problem is that the protocol should give users a solid foundation to build complex things with as much security and transparency as possible, but it needs to be simple in order to minimize the number of exploits.
Just my 2 cents.