Wearing a mask cannot increase your viral load. A virus differs from a bacterium in that it cannot replicate outside the human body. It cannot replicate in a mask. It can catch virions that you breathe out, but you've got loads more inside your lungs if you're doing that. When the virus has just arrived in your nose or mouth or eyes (nose seems to be its easiest point of entry), it doesn't replicate very well (and in fact, your mucosal membranes and stomach acid, etc, will kill a bunch of it). But if you breathe it in through your nose or it gets into your sinuses from your eyes, then it gets into your lungs and really begins to replicate there.
There are some comments on this study that are relevant:
Face Masks Against COVID-19: An Evidence Review
It's a very long article, but the upshot is that you should treat a mask like a contaminated surface. The mask will trap and destabilize some virions, other virions will just sit there until you touch them, then if you touch your nose, you'll reintroduce them. The idea of a mask is that it is supposed to trap many virions in its fabric, disabling the ones you just exhaled from re-entering your lungs and joining the others to replicate. You're still infected in this scenario, but the mask isn't making it worse, it's giving a tiny bit of protection.
When you take the mask off, now you've got virions on your hands (but the mask has way more virions on it - toss it or wash it or expose it to heat/UV); wash your hands, etc. Even if you were to forget and touch your face after taking off your mask (which potentially has virions on both the inside and the outside if you're in an environment that's infected), you won't be getting tons and tons of virions on your face (I always wash my face when I wash my hands, but that's being a tiny bit paranoid).
If all is working properly your exhaled aerosols and droplets (if you cough or sneeze) stick to the mask and cannot replicate, you could breathe in a few more of the ones you just exhaled, but that would happen in any case (and you'd be infecting others).