I think rubbing would cause more damage than patting (from the friction). I always put a towel over my shoulders and dry my hair on top of that. I get the extra water off by patting. For me, patting is just gently squeezing one area, but keeping my hand in one place (no rubbing).
My hair isn't quite long enough to dry inside a towel, but the towel helps soak up the excess water that way.
Bookmarks