Police were called twice in two days after children were seen on the roof of the old Thorntons factory in Belper.
Fifteen youngsters were spotted on top of the derelict building on Derwent Street on Friday and six were seen there on Saturday lunchtime.
Belper resident Ian Rumboldt said: "The roof is probably rotten and you are talking about death or paraplegia if someone falls through or off it."
A Derbyshire police spokesman said: "We attended the scene but couldn't see anyone."
The site, which stopped production in 2004, was devastated by a gas explosion in 2013.
Last year, Amber Valley Borough Council approved plans by Spenhill Developments to demolish the old factory and build 107 homes.