From pastime + -er. First use appears c. 1608 in a letter by John Harington.
pastimer (plural pastimers)