One of the biggest dangers with a gathering this large is the spread of disease.
Think about it. You have people coming from every part of the world, carrying who knows what germs, bacteria, and viruses with them. In the case of a serious strain of influenza (think H1N1 or Avian flu), any kind of outbreak could be deadly.
Germs and viruses can spread very easily here. To complicate matters, many parts of the world don't have high levels of health education. In some countries, it's common to sneeze or blow your nose right onto the ground. In Mecca, where people are generally barefoot in holy sites, it becomes a recipe for disaster.