It depends on what in the jewellery has caused the reaction. If you think the metal might be nickel, a reasonably common allergy, the rash should fade in a few days, providing you stop wearing the culprit. And try not to touch or scratch it. If you haven’t reacted to a metal previously, it might be worth getting an allergy test done to confirm what has caused it - it could be perfume on the jewellery.
To speed things up a bit, you could try a colloidal oat treatment directly on the rash. Having said this, there is some, small, anecdotal evidence that oats may actually contain enough nickel to cause a reaction in people vey sensitive to it. Actually, it is very unlikely to contain more than 2 ppm (parts per million) and I can find no trials with any significant reaction using less that 10 ppm - and the oats will be diluted in whatever liquid they are mixed with to make a paste - so… it is highly unlikely that oats will make it any worse! They are very soothing, gentle and easy to find. Alternatively, aloe vera can be good for the itching and the soreness. You could also use a calendula, chamomile, cornflower, evening primrose, lavender or rose hydrosol as these have also reduce the itching and soreness from a variety of causes. Do make sure you have no allergies to the plants and please only use actual hydrosols, not water with added essential oils. (Look on the ingredients list for a polysorbate or any ingredients, other than a simple preservative). If it has added oils, these could further irritate the area.
If it’s not nickel - let me know!