Number of strings generatable I really do not know how to start with the conditions of this problem:

A H64 string is a string with 64 characters, each character must be
  choosen from 16 hexadecimal characters (0 - 9, a - f). However, no
  character can appear over 10 times. How many H64 strings that can be
  generated from the above condition?

The normal solution to generate a string with 64 characters is 16^64, how can we eliminate the cases with 10 times appearing characters?

This is not a full answer. But I think there are enough hints to work it out.
Let $A_k$ denote the set of such valid H64 strings that make use of $k$ distinct hexdigits. By pigeonhole principle, the given condition leads to the constraint that $k\ge6$. For $k\neq l$ clearly, the  two sets of strings $A_k$ and $A_l$ are disjoint. 
So the final answer is  $\sum_{k=6}^{16} |A_k|$.
Let me indicate how to find the cardinality $A_k$. First you have to decide on which $k$ symbols out of the $16$ are going to be used in a single H64 string. This shows that $|A_k|$ is $16\choose k$ times the number of H64 strings that can be formed just using $1,2,\ldots, k$.
You can relate this to the count of monomials in $k$ variables of total degree $16$, but the difference is that $x^2yz$ and $xyxz$ are to be counted as different monomials.  At this stage I am not   able to come up with a good approach to proceed further.
