This is of course just my personal opinion, but sometimes colored gravel can get on your nerves. Something netural is usually better. Of course if you are not lazy (like me ) you can replace it any time.
the size of the gravel also has alot to do with bacteria, and waste colection, if you know about surface area, the smaller the gravel, the more good bacteria you have, and the larger the gravel, the easier it is for waste and un eaten food to sink into it, and its also harder to vaccum clean, so just think about those little tips, i dont think color affects the fish, but it can certainly affect your perception of them, i like really dark colored gravel/sand. its a better contrast to the colors of the fish, but some people prefer really lite colors. but i really boils down to what YOU like, so just go with that.