I've picked up 5 sources that think that my server is their personal "ping toy."
I can't figure out if you are another one of those people who thinks that all ICMP packets are evil, and block them whenever you see one, or if you actually experienced some abuse.
Responding to an echo request with an echo reply is a mandatory part of IPv6, and some protocols rely on this, most notably Teredo can't communicate with a host that doesn't respond to echo request. If you block the request, you'll never know if it was part of legitimate traffic.
A constant flow of say one small echo request per second may be unusual, but if you feel that amount of traffic will have a negative impact on your systems performance, then you are doing something wrong. Blocking traffic based on such small numbers is going to cause you more problems than it solves.
There is a huge margin between the normal amount of echo requests, and the amount it takes to slow down a system. Since you didn't mention any actual numbers, I can only guess at which of the two the actual number was closest to.