By allowing these types of materials to build up this cause 2 things to happen to your gutters, one of those is ruining the look of your gutter by allowing the dirt to build it it can drop the level of look on your house.
The second one is damage, the damage that can occur when leaving a gutter unchecked or cleaned out, its leads to gutters spilling over or cracking under the weight of the dirt, leaves and moss in the gutters.