Close to Bywaters in Velen, I ran into two map icons shaped like spoked swirls. I didn't find anything there. They only show up in the minimap, not the full map. Anyone know what those are for? Maybe something underground?

Yep, it's the portals, underground.
Dunno if you did it already, but, for example, the quest with Keira Metz, where you open a bunch of portals.

Yeah, I have done the Keira quests; just forgot what the icons look like. Are these particular portals from a specific quest; do you know how to get to them?

Depends, try and look around, maybe you'll find an entrance to an underground cave or something.
Or probably they are from a quest down the line, or one that you already did. I can't recall what quest every portal is from, or if you can get to it just by exploring.

One thing i can tell you, if it's icon is showing on the map, then it means it was visited before. They have to be activated first before they show.