Definitly normal
Just cause it's physicaly and mathematicaly more difficult to calculate.
Reflect/refract are the hardest thing to render, so if u blur it (reflection/refraction glossiness), that will be more and more difficult.
You can tweak reducing the maxdepth and the subdivs instdead of quality.
U can try that for your reflective/refractive materials :
reflect/refract glossiness : 1
maxdepth : 5
subdivs : 8
reflect/refract glossiness : 0,8
maxdepth : 3
subdivs : 12
reflect/refract glossiness : 0,6
maxdepth : 3
subdivs : 20