You are right, though, that might conditions can also be at work. That should be obvious just from observing the light patterns each day. Is one side getting significantly more light?
Sounds to me like the ones on the smaller side might also be getting less of the available water and nutrients than the larger. Not just the clay can be a factor, but tree roots (which go a long distance from the trunk in many cases, especially in older trees) can outcompete those of herbaceous plants and starve them for what they need to grow to full size. Do you water well?
Was the clayey soil improved with lots and lots of compost before planting?