Here is a great tip for how to clean this "ugliness" off of your Rough-tex. This comes from BENJAMIN ROBERTS -- the very capable Asst. Aquatic Director at The Ohio State University and the person who is charged with keeping one of the finest diving facilities in the world in top working condition.
Take a piece of PVC pipe (about four feet long and three inches in diameter) and wrap duct tape around the PVC Pipe with the sticky side facing outward. Then, just like a large lint brush, roll the sticky PVC pipe back and forth across the Rough-tex and it picks up most of the debris that has attached itself to the Rough-tex. (Make sure the Rough-tex is dry before doing this!)
Afterwards, do a quick power washing of the Rough-tex to remove any remaining debris or oils or anything else that could make the Rough-tex lose some if it's non-skid grip.