Welcome!
Nope! The ink might stain the rubber a bit, but as long as the design is clear with no chunks of dried ink in the small crevices to clog the image after you clean the stamps, you should have lots of fun stamping on your scrapbook pages. (shampoo or glycerine soap & a baby toothbrush are good for cleaning stamps if you don't have a good stamp cleaner on hand)
A mild warning...unless a dye ink is labeled 'light' or 'fade' resistant, it will eventually fade a bit. Especially if left in direct sun. Of course, direct sun isn't good for cardstock or photos either.

So a bright pink flower stamped today, may be just a medium pink flower years from now & softer colors (i.e. pastels) made fade out quite a bit.
Pigment inks are more fade resistant than most dye inks & can be found in pen form & there are several dye inks on the market formulated to be more light resistant than regular dye inks. A few of the better scrapbook safe dye inks are are: Ancient Page from Clear Snap, Archival Inks from Ranger & Momento from Tsukineko (now availble in markers). I know the other members will add info about other inks & good advice, 'cause that's just how we are around here!
P.S. Check the labels of any inks (pads or markers) for the words 'archivally safe' or 'archival quality' or something along those lines. Acid free alone isn't good enough as a product can be considered acid free because it's highly alkaline, which is just as harmful to photos in its own way as highly acidic materials can be.