Quantitative Prediction on the Impact of Defects on the Elasticity of Polymer Networks