Preparation is really helpful. Walking in may be less threatening if you choose less populated/less busy times to shop, gradually working up to busier times, or if you are accompanied the first few times you go. Before leaving the car, practice some box breathing, then calmly go into the store.
It helps to have a grocery list made in advance. That will eliminate the worry about what you may be forgetting and will help you organize your navigation inside the store if your list is made according to store sections.
Have your shopping list at the ready. If the store noise/music bothers you, how about wearing a walkman? I see runners and athletes wearing them everywhere. You could have a relaxation tape in yours or a motivational tape or even one you make yourself with coping strategies on it.
Some people like to begin by buying less than 8 items. The next trip can be nine items and so on.
If you can choose the time you go, then you can choose the express lane and then work your way up to standing in the regular checkout behind one person, then two, etc.
Before your thoughts threaten to make you run out of the store, have some planned reinforcing self-talk ready and say it, like at the middle or end of every aisle you visit. Remember that this will not kill you or necessarily bring on an attack.
Sometimes the stores are crowded. But even in such situations, there ususally is a spot where one can stop and collect one's thoughts in the middle of an aisle. Regroup before the end of each aisle before leaving it and taking on the next.
Keep practicing positive reinforcement and box breathing. If having enough money is a source of anxiety, you can bring along a calculator. I've seen lots of people doing this and adding up items as they go along.
Buy yourself a little reward too.
Hope this helps you make your own baby steps plan.