The big concern is blue and then brown asbestos, thats the stuff they used to lag pipes, make gaskets and is sometimes in comercial buildings with, but then its only if its airbourne that its any concern. your not going to drop down dead from walking through an old factory with some pipes covered in the stuff!
The large majority of whats found in building construction is white which is the least harmful of the lot.
Im not saying its harmless, I will not expose myself to the stuff if I know its there, but its not something ill lose sleep over if I see a chunk of it laid on the floor.
Just be aware that when its airbourne, its not something you want to be actively inhaling. It can get pretty bad when in a bunker thats been smashed up and set on fire or somewhere similar.
If im out and its very suspect or known that theres asbestos, ill wear a mask. P3 is more than adequate, especially if its a proper face mask. I would be happy with P2