Finding the mode of the negative binomial distribution The negative binomial distribution is as follows: $\displaystyle f_X(k)=\binom{k-1}{n-1}p^n(1-p)^{k-n}.$

*

*To find its mode, we want to find the $k$ with the highest probability.

*So we want to find $P(X=k-1)\leq P(X=k) \geq P(X=k+1).$
I'm getting stuck working with the following:
If $P(X=k-1)\leq P(X=k)$ then $$1 \leq \frac{P(X=k)}{P(X=k-1)}=\frac{\binom{k-1}{n-1}p^n(1-p)^{k-n}}{\binom{k-2}{n-1}p^{n}(1-p)^{k-n-1}}.$$
First of all, I'm wondering if I'm on the right track.  Also, I'm having problems simplifying the binomial terms.

You are on the right track (except for typos in the notation now corrected, see @Henry's Comment and my response). Express binomial coefficients
in terms of factorials. Some factorials will cancel exactly. Others
will have factors in common: for example, 10!/9! = 10.
For a simple start, you might try the case where $p = 1/2.$
Answer: The mode is at the integer part
of $t = 1 + (n-1)/p,$ if $t$ is not an integer. For integer $t,$ there
is a 'double mode' at $t-1$ and $t.$ 
Examples: Below are three examples that illustrate this formula
(4-place accuracy):
 n = 2;  p = 1/2;  t = 3,  mode at 2 & 3
  k  :     2      3      4      5
 p(k):  0.2500 0.2500 0.1875 0.1250

 n = 2;  p = 1/3;  t = 4,  mode at 3 & 4
  k  :     2      3      4      5
 p(k):  0.1111 0.1481 0.1481 0.1317

 n = 2;  p = .4;  t = 3.5,  mode at 3
  k  :     2      3      4      5
 p(k):  0.1600 0.1920 0.1728 0.1382

Note: Wikipedia and many advanced texts use a different form of the negative
binomial distribution where only failures before the $n$th
success are counted. Hence $X$ takes values $0, 1, 2, \dots.$
For the mode according to that formulation, see Wikipedia.
(It is noted later in the article, to add $n$ to the mode
given at the head of the article for your formulation.)
