If your mains (the power coming from the wall) is not grounded, there will be 3.5mA of power looking for a ground. When you touch a metal part of the PC, you become that ground. While not life threatening, using a non-grounded outlet is not ideal. We would suggest making use of a grounded outlet, or using a power strip which is grounded.
Power supplies use “Y” capacitors connected from the neutral to ground as part of their integral EMI filter. These specially rated capacitors provide a low impedance path to the ground for high frequency noise to reduce EMI.