When a capacitor is charged and then the supply is disconnected charge present on it becomes constant (law of conservation of charge)
Initial capacitance C=kε0Ad (with slab)
Final capacitance C=ε0Ad (slab removed)
We know Q=CV
As C decreases and Q remains constant V has to increase.