Assume this equation has distinct roots. Prove $k = -1/2$ without using Vieta's formulas. Given $(1-2k)x^2 - (3k+4)x + 2 = 0$ for some $k \in \mathbb{R}\setminus\{1/2\}$, suppose $x_1$ and $x_2$ are distinct roots of the equation such that $x_1 x_2 = 1$.
Without using Vieta's formulas, how can we show $k = -1/2$ ?
Here is what I have done so far:
$(1-2k)x_1^2 - (3k+4)x_1 + 2 = 0$
$(1-2k)x_2^2 - (3k+4)x_2 + 2 = 0$
$\to (1-2k)x_1^2 - (3k+4)x_1 + 2 = (1-2k)x_2^2 - (3k+4)x_2 + 2$
$\to (1-2k)x_1^2 - (3k+4)x_1 = (1-2k)x_2^2 - (3k+4)x_2$
$\to (1-2k)x_1^2 - (3k+4)x_1 = (1-2k)(1/x_1)^2 - (3k+4)(1/x_1)$
$\to (1-2k)[x_1^2 - (1/x_1)^2] - (3k+4)[x_1 - (1/x_1)] = 0$
$\to (1-2k)[x_1 - (1/x_1)][x_1 + (1/x_1)] - (3k+4)[x_1 - (1/x_1)] = 0$
$\to (1-2k)[x_1 + (1/x_1)] - (3k+4) = 0$ or $[x_1 - (1/x_1)] = 0$
$\to (1-2k)[x_1 + (1/x_1)] - (3k+4) = 0$, $x_1 = 1$ or $x_1 = -1$
Since the later two cases violate the distinct roots assumption, we have:
$(1-2k)[x_1 + (1/x_1)] - (3k+4) = 0$
This gives:

The answer is $x_1 = \frac{5-i\sqrt{39}}{8}$ or $x_1 = \frac{5+i\sqrt{39}}{8}$. How do I get that (without Vieta's)?

The problem is find $k$ not $x_1$ isn't it? If so take the equation you arrived at before you started doing computer algebra:
$$(1-2k)[x_1 + (1/x_1)] - (3k+4) = 0$$
and multiply by $x_1$ to get
$$(1-2k)x_1^2 - (3k+4) x_1 + (1 -2k) = 0$$
but you know that 
$$(1-2k)x_1^2 - (3k+4) x_1 + 2 = 0$$
so you must have $1 - 2k = 2$, i.e., $k = -\frac{1}{2}$. (Using the Vieta formulas which just amount to $(x - x_1)(x - x_2) = x^2 - (x_1 + x_2)x + x_1x_2$ for a quadratic seems a much simpler approach to me.)
