Some time ago I was working on project where I needed to query items from several lists and document libraries (each list and library contained 200+ items). I needed to display Title and Url fields from lists and file name and file url from document libraries.
My first code worked too long and I've started to think about increasing the performance of queries.
And here are the things that helped me to increase speed in several times:

  1. Use <ViewFields> inside query instead of Include(...) inside ClientContext.Load()
  2. Use FileRef and FileLeafRef fields instead of File field (which is an object with several properties in it).

That's all. These two changes helped me a lot. And the customer is happy about the performance they have.
Have fun!