Am I correct in thinking you have not installed any of the tubes yet? And also haven't installed any separator (colon) LEDs?
If so then I don't think anything is wrong. The test sequence is RED - GREEN - BLUE - COLONS - RED - GREEN..... so the "kind of dims before starting over" may be the step where just the colons are supposed to be lit. Re-read Page 13 of the construction manual carefully.
In order to move forward iot would be helpful to have at least one colon LED, the tubes (or at least one tube) and the pushbutton connected.
The 7805 is going to run warm, but not smoking hot, no matter what. In test mode it will be quite warm because the RGB LEDs are being lit at full power. In normal operation it's likely you'll use LED settings that aren't so bright (= less power demand) because those LEDs are REALLY bright.
I think the 0.9V thing is a measurement error. Try VIN to GND,(or IOW D1 cathode to GND). If you just read across the diode you will measure only the diode drop which is likely to be 0.7 - 1.0 volt.
Look into it later when the dust is clearing off the crater.