Well I figured it out. Apparently the BCM3510 won't lock onto an
Annex A 64QAM or 256QAM if it is stuffed with MPEG null packets. I
modified my mpeg block to send a 0x47 sync byte, followed by 187 bytes
of PRBS15 sequence and now the BCM3510 demodulator is locked and
happy.

To me this seems like a bug in the demodulator, why should it care
what is inside the mpeg packets? The next step in the modulator is a
randomizer, so there was already plenty of randomization in the bit
stream being fed into the interleaver - byte to symbol mapper -
etc. . . Plus the BCM3510 was happy locking onto Annex A 16QAM,
32QAM, and 128QAM with MPEG null packets.

Anybody ever hear of anything like this?

Thanks
Brett