These Edoites are likely coming from and going to the Ekôin temple fair (kaichô) which is advertized on the banners in the background. The main hall of the Ekôin itself is visible on the left. Choose to view the kaichô or follow sumo wrestlers to the so-called benefit tournament they are giving for the temple.