I've left chargers on batteries overnight countless times. These batteries were installed in cars, boats and lawn tractors, or sitting on a rubber mat in my garage or basement.

I always make sure the alligator clips are secure. (Note: always install the negative side - marked with a minus sign - before connecting the positive terminal, which is marked with a plus sign. When finished, remove the positive connection before the negative.)

I would guess that a loose negative clip is the cause of many fires.

A quick google search confirms that unattended charging is not recommended due to fire hazard. Guess I'll have to change my ways.

Most battery chargers for specific devices don't charge at a high enough rate to present any danger. General use charges can be a problem because the can be misused. The most likely to be a problem are chargers for battery powered tools because the batteries are high capacity and charged at the highest possible rate. Anything that's newer than 10 years old is likely built with sophisticated shut off technology. It's so cheap to build in and lengthens battery cycle life so as not to piss off customers. A good warning sign is if the batteries get warm while charging.
If you want to provide an added bit of safety and save on electric bills put all your chargers on a power strip and turn it off when you're not home.
You did make a smart move by unplugging them until you need them. Anything with a wall wart or remote is drawing power 24/7. Unless you have electric heat or need AC year round you could cut your utility bills by about 10%.