What is the max ddr2 memory size and speed of Intel Core Duo E8500 and Quad Q9650?

That's a perplexing question because with the socket 775 platform the Memory Controller is on the motherboard..... and it's known as the Northbridge chipset. Usually the RAM runs at 800mhz out of the box. If you have a motherboard with a P45 chipset then you can manually set the RAM speed. Motherboards with the P45, P43, and X48 chipset are capable of 16gb of RAM whereas cheaper chipsets like G41 are only capable of 8gb of RAM.

Max memory size depends on the motherboard since the Northbridge chipset is on the motherboard.

With newer Core i processors the Memory Controller is part of the CPU.

That REALLY depends on the motherboard that you're fitting those individual CPUs into, my friend.  The motherboard chipset is what will determine which speed RAM can be used, and how much RAM can be installed.