Handling metal lead is very safe actually, the body surface doesn't absorb metallic lead, but organic lead compounds can be absorbed dermally. Since you said you are exposed to tubing and fittings (and more importantly solder) with high aluminum and lead in your body the elevation is probably a result of exposure to solder vapors and/or fine metallic particles through inhalation and incidental ingestion. Also if you sand paint off walls or are around that is another easy way to get lead exposure.
Most of the environmental concerns for lead sinkers and the like is not because of fish, but birds. Many birds seek out and swallow small round stone to aid in digestion, so a nice round sinker or shot looks pretty good for that purpose. The grinding and the stomach acids causes round the clock exposure for these birds, often times leading to neurological damage and death as well as negatively affecting reproduction.
The same thing goes for lead in our daily lives, if you aren't exposed by work it really isn't a problem to the average adult. Children are a different story: as anyone who has been around kids knows, they put stuff into their mouths and more than parents would like to admit, eat some strange stuff like dirt. Children are also closer to the ground than adults and more susceptible to dust borne lead from flaking lead paint, contaminated soils. In addition, their small body size and rapid growth can cause what would be essentially harmless exposures for adults to have noticeable impacts on growth and development.
If you're thinking of lead pipes and the downfall of Rome, they added lead acetate (an organic lead compound) to wines to add sweetness. They called it "Sugar of lead." They also used it in tableware, cosmetics, glazes and other applications. Being around lead all the time like that is similar to what the birds who swallow lead go through, but people were still not dropping dead all over the place from lead poisoning.
Not that banning lead from certain applications is a bad idea, bullets add way more lead to the environment than fishing, lead paints and leaded gasoline, but I like facts and not scare mongering play the part in deciding legislation and regulation.
If I were you I'd worry more about people driving distracted than lead weights. If you are concerned about work exposure send me a msg detailing what you do and I'll run through probable exposures with you.
edit: sorry if I sound like an ass, I just went through years of schooling on bio, public health and toxicology so I like to use the info. It makes me feel like I got my monies worth