I was in a situation much like this a few years ago. If you want to go after lost benefits, start by filing for unemployment. Determining Employee/Contractor status is one of their steps to determining eligibility. If you've been filing taxes as a contractor, you may be eligible for some IRS refunds. Contact the IRS and file an SS-8 ASAP because another year will fall off if you wait to Jan 01. (It takes at least 6 months for an official determination but the date is based on when you filed.) Based on the information provided above I believe the determinations would be that you are in fact an employee.

Employee/Contractor status is determined by who has "control" while you are working not an agreement upon hiring. The less say you have regarding when/where/how you work the more likely you are an employee.

Additional Reading:
https://www.irs.gov/businesses/small-businesses-self-employed/independent-contractor-self-employed-or-employee