When the US Constitution came into force in 1789 the Federal Government was established in New York City. The Senate met on the second floor of Federal Hall. In December 1790 it began a ten year stay in Philadelphia - in what is now the upper floor of 'Congress Hall' - a building close to Independence Hall.
The Senate chamber - and the adjacent committee rooms - can be visited by the public.
A list of the first Senators and Representatives can be found here. Of the Senators who served in the First Congress - William Paterson (NJ) William Grayson (VA) and John Walker (VA) - only served in New York.