The clear gravel under the deck is OK. However, if the soil under it is topsoil it needs to be removed.
The trench should be filled with compaction gravel and then mechanically compacted NOT WHEEL ROLLED.
As for the patio infill and its composition, there are many schools of thought on how to go about this task. Some may say to use clear stone most of the way up your retaining wall to allow for drainage. Then use a 4-6 inch lift of compaction gravel on top of that. Between the 2 layers you would place a geotextile and also on the sub-soil.
This school is based on the assumption that pavers allow a great amount of moisture to pass thru their joints and peculate into the base material below.
Another school of thought is using all compaction gravel as in a traditional paver installation with a geotextile underneath.
This philosophy is based on the knowledge that pavers are not as free draining as most people think.
Currently the ICPI
is creating a specification on raised patios. Their construction committee will be discussing it at the summer meeting next month.