This is fairly trivial but I wanted to document the steps carefully.
The code is here and I flashed the ATTiny85 using the steps I referred to here.

The flash pins are still connected though only VCC and GND are necessary to run the program because they supply power. I have 5V running across them which is coming from the Teensy which, in turn, gets it from the USB.

So save confusion I am going to refer to the ATTiny pin numbers with a #, so #4 is GND and Arduino Pin 3 is #2 etc.

So the speaker is connected across #3 and #4 (Arduino Pin 4 and GND). I tried several speakers ranging from 9 ohm to 47 ohm. Not a lot of volume but quite audible. There were no resistors or capacitors etc.

Once running the program plays a 7 note melody over and over until you get annoyed with it and turn it off.

But there is a critical step I missed the first time around. There are several implementations of the ATTiny core for Arduino. Not all of them support the tone() function because it is tougher to do on an ATTiny (so fair enough).  Anyway the one I used is here. That also lists a bunch of other implementations so there is a lot of choice. The readme tells you how to add it to your Arduino directory.

Once it is installed you do need to pick the right ATTiny85 option in Tools->Board. There are several and the one I found that sounded right was “ATTiny85 @ 1 MHz (Internal oscillator, BOD disabled)” The others are oscillating at different speeds which means the clock doesn’t run at the speed you want and the melody is scrambled.

I have the melody working on #3 just fine although it is not documented in the diagram as a PWM pin. In fact #2, #3, #5-7 all work. This may be because the Arduino core I am using is not actually using PWM for the tone(). Not sure. It works anyway. The comments in the core code match the diagram above , ie #5 and #6 are PWM and no others.