
My First Year as a Freelance AI Engineer - polm23
http://masatohagiwara.net/202002-my-first-year-as-a-freelance-ai-engineer.html
======
mhagiwara
Wow, it's nice to see my post on the top page of Hacker News.

Note that I wrote this post back in February 2020 and there are a couple of
things that I would like to add:

\- Due to high demand, I increased my rate to $250/hour + some fixed monthly
fee in April. I didn't drop any single client :)

\- I'm seeing very little impact from the coronavirus. I have a client base
spanning between Japan and the US in a little-impacted industry (education).
Don't put all your eggs in one basket.

\- I would strongly encourage everyone who's considering making a leap to read
"The Win Without Pitching Manifesto" [https://www.winwithoutpitching.com/the-
manifesto/](https://www.winwithoutpitching.com/the-manifesto/)

\- Due to a sheer volume of my incoming emails I can't answer all of them, but
do let me know if you are interested in working with me!

~~~
mhdhn
I don’t get what “NLP/ML for Asian language processing and language education”
work is. Can someone explain a bit more. What kinds of companies do this, what
kind of businesses?

~~~
polm23
An example of "language education" is Duolingo, where the author used to be an
employee.

"NLP/ML for Asian language processing" covers anything that has text in
CJK/other languages. I work with Japanese and lots of things taken for granted
in NLP pipelines require entirely different approaches specifically for
Japanese. For example, there's no spaces, so word tokenization is actually a
complicated issue.

------
hash872
As a business owner & non-engineer who sometimes uses freelance software
engineers like OP, it's fascinating to see so much pushback to the guy's
hourly rate. On Hacker News!

People are so hung up on how much the guy gets per hour, as a business owner
all I'm thinking about are the _total costs_. There are lots of cheaper
developers who would take longer to get the job done- how have I saved money
by hiring them? And that's before getting into quality of work. I think many
commenters are imagining OP as a contractor who works 40 hours a week for his
clients, 9 to 5. The whole point of a consultant is that they're project-
based, it's not an ongoing expense. I would certainly hire OP for a 20-50 hour
project, say, and then an hour here or there to fix bugs and answer questions
as needed. It's not a huge total cost, I don't care what the guy gets paid per
hour. I literally just look at it like 'overall, this ML/AI project cost me
$10k, and delivered x amount of value'.

If HN commenters think $250 an hour is a lot, wait till they hire accountants
or- god forbid- attorneys :) I bet OP is much smarter and delivers more value
than my attorney that charges $400 an hour for simple contract review, and
breaks out specific line items for answering my e-mails and 10 minute phone
calls with me....

~~~
tcbawo
When I notice people pushing back on hourly rates, I think of a story a friend
told me about an expert locksmith. He could unlock a door in under 30 seconds,
but he often fiddled around a bit so his customers would feel better about
having spent $200. He was very good, but if he made it look too easy, they
would complain or give him a bad review. I try to never begrudge someone that
knows how to add value and set a marketable price. If you're consistent,
reliable and add value, they deserve it!

~~~
Rastonbury
I'm not saying it's easy but $200 is a lot. Hypothetically purely based on
opportunity costs, the AI freelancer could afford spending an hour trying to
unlock his own door, someone like me I could spend like 6 hours buy a lockpick
and watch a YouTube tutorial, or maybe easier to spend the next hour looking
for a cheaper locksmith!

~~~
purerandomness
That is only possible because you didn't put a price tag on your own time.

Similarly, a company's CEO could say "Hey, I could simply go to university
myself and learn everything about NLP and solve my NLP problem and save
10.000$ for this contractor! Yay!"

That's only scalable up to a certain point, and that point is usually reached
when you realize that you need employees.

------
ganstyles
Super cool, I've been thinking a little about this for CV rather than NLP. One
thing that gave me the idea is I (the company I worked for, at the direction
of me) engaged an outside expert on a particular ML domain. It wasn't a
freelancer, but just someone at a major company who did some cool stuff. I
approached them, told them about what issue we were having, and engaged them
for a 50 hour contract @ $500/hr. Completely worth it, but looking at your
rates maybe we super overpaid. Not that it matters, but just an interesting
data point and got me thinking about freelancing purposely if those are the
rates. Thanks for sharing your experience.

~~~
nvilcins
New to freelancing, and while I get the hustle and pricing according to the
value for the client, it's still hard to wrap my head around people charging
500$/h+. Like, what is the value an AI contractor can deliver in ~50h that
easily justifies spending 25k (and without much prior knowledge about the
product, systems in place, etc.). Honest question, trying to understand the
game.

~~~
ganstyles
Imagine developing a core competency for your company where it's complicated
enough that you're not sure what route to take. Also assume you have a team of
maybe 8-10 ML engineers who all make what ML engineers make. It makes so much
more sense to pay $25k to get direct knowledge of systems and techniques
versus your team spending a bunch of time exploring different
products/methods/algos to find something that might work in the end, or might
fail in a few months. $25k is like 5 MacBooks, hardly worth thinking about
versus being able to get experienced direction from someone who has done what
you're trying to do and saving your team literally hundreds of hours of
exploratory work. It's not really a game, it's good sense. Yeah given the covo
here I probably could have negotiated them down, but that's my time doing that
and then things have to be vetted with outside counsel and the CFO, also time
wasted, versus just telling them this is what the person charges, we need them
for 50 hours, "authorize the charges please." And at what cost, saving a few
thousand dollars?

~~~
mlthoughts2018
How do you develop a business connection and market your service as being more
valuable that the wide abundance of free materials, engineering blogs,
research papers, etc., in this space?

Just for example, I run a team of 10 machine learning engineers at a large
ecommerce company. We mostly do NLP and computer vision, some time series
forecasting.

I cannot imagine ever paying anything close to $25k for consulting advice,
that’s just bananas to me. We recently purchased licenses to use the data
annotation tool prodigy from the spaCy creators at explosion.ai. That was
~$4000 and the decision whether to build our own data annotation system or not
was excruciating, involved all kinds of business documentation, RFCs,
approvals, NDA processes, etc. It was deeply non-trivial to procure that, and
building our own was a very serious option we pursued with tech specs and
prototypes and everything.

Spending 6x that amount for _advice_ about NLP, which practically grows on
trees today, is just totally unrealistic.

It makes me suspect the real target customer for you is not companies with
actual ML engineering teams or ambitious data-driven projects, but more like
someone looking for McKinsey-lite. Some place that has no serious ML use case
beyond drop-in pretrained models and sees $25k as the cheaper path to rubber
stamp certification that dissolves internal political feuds. Most likely just
selling super cookie cutter NLP models as if they are advanced and represent
some sexy leap forward for a company with a couple junior data scientists.
Algolia or just some drop-in Elastic tfidf search is more than enough for
these companies. Spend the $25k on an intern who can tell you anything you
need to know about neural network frameworks.

In reality, the 4-5 ML engineers you already hired are very likely _more
knowledgeable_ than the freelance consultant you might hire. They can tell you
much more about state of the art and simultaneously know the specific
integration path in _your_ company’s web service and data ecosystem. Those
folks won’t be wasting time prototyping - they would be pursuing a _more
efficient_ way to get the answers you need than advice from a freelancer, even
if that freelancer was Bengio for pete’s sake.

I just cannot see the value prop here except for the usual story of paying for
consulting as a virtue signal / credential / politics kind of thing.

~~~
ganstyles
I just really don't know what to tell you, I'm sorry. We saw the value,
pursued the person, and had an engagement with them effectively consulting for
50 hours. There was no justification to anyone outside of our team, other than
getting the very quick approval, and the consultant worked with our team
directly and not to anyone outside the team. There was no internal politics at
play.

And again, this wasn't a "freelance consultant" in the sense of the original
story posted to HN, but an accomplished person at a very respected company who
was able to secure the approvals from their side to help us out. This was
novel/niche work for which "drop in pretrained models" don't really exist or
apply.

Frankly to judge and belittle someone else who is just sharing their
experience by saying they're not doing serious work and don't have serious
customers is very rude.

~~~
mlthoughts2018
It’s not rude at all to highlight how this warrants huge skepticism. “Just
calling someone up” to spend $25k+ on “advice” and claiming it’s justified in
time saved of your existing staff not flailing around to research the same
topic is _very_ weird. That doesn’t line up with any professional experience
I’ve ever seen in any machine learning team at Fortune 500, startup, finance &
academia jobs. _Maybe_ for appearance fees to have a big name researcher come
and give a company presentation or keynote. _Maybe_. Absolutely not believing
it for ad hoc freelance consulting based on someone having some NLP projects
on GitHub, conference presentations or publications, or gigs at “fancy” AI
employers, and I find it’s worth making this comment so other readers can
consider how extremely shaky that premise is.

~~~
arbitrary_name
Just to provide another perspective : I've consulted for 20+ companies.
Spending money for outside advice or actual work products is extremely common
and generally worthwhile if done right. There's a reason why consulting is a
multi billion dollar industry.

~~~
mlthoughts2018
I disagree. Consulting is fashion signaling. It’s a huge business because it’s
about buying and selling political status in the context of company decision
making. It offers value from social signaling, not from strategic facts that
demonstrably yield improvements to success metrics.

------
forgingahead
Great article! The most important part for others reading this:

 _Narrowing your niche down attracts specific types of clients who have
specific needs that few people in the world can solve._

 _My expertise is NLP /ML for Asian language processing and language
education. When defining your specialty, I think it helps if you define it in
terms of the industry, not in terms of an ML stack. People look for, e.g., “AI
solutions for healthcare” and “text analytics for finance,” not for “GANs” or
“Seq2Seq models.” You need to be willing to learn a very wide range of ML
techniques and models, from simple regression to GANs and RL, no matter what
industry you work in._

In other words, _define your niche in terms that make it easy for someone with
budget authority to say "yes" to hiring you_.

This is worth noting also because if you accidentally cross over into talking
with recruiters or other gate-keepers to the real budget authority holders,
you'll be misled into talking or padding your resume with technical terms that
are not the key value drivers for why you'd get a nice-paying contract.

------
damon_c
One interesting thing about being freelance during times of economic
instability is that the "steadiness" metric is inverted when you have multiple
sources of income compared to relying on one company.

------
aantix
LLC? You’re leaving thousands on the table.

Switch to an s-Corp.

30% of the gross will be paid as a salary, as an employee of the s-Corp. you
must pay social security, etc on that.

The other 70% can be taken as a dividend distribution As the owner of the
s-corp. Thats not subject to social security, etc.

Will save you ~15% in taxes.

at your level, it’s Well worth the cost of the cpa to handle the payroll,
quarterly, etc.

~~~
mhagiwara
I do have a CPA and we did an S-corp election for 2019 tax return. This is
something I didn't get to mention in the post because we hadn't filed the tax
yet.

~~~
aantix
Maybe convert it in a a future post.

I don’t think devs realize that a 1099 costs you money.

------
mfalcon
What about client's valuable and private data? I've always thought it'd be a
big problem for freelancing as an AI engineer.

~~~
mhagiwara
This has never been a big problem for me. Some clients have strict policies
(e.g., only allowing you to work on their servers) while others don't care as
much. In some cases, as a freelancer, you only work on a "proof of concept"
prototype assuming you have access to their private data. It is their FTEs job
to actually train/implement/deploy this prototype using their private data.

~~~
mfalcon
How can you work on a "proof of concept" prototype without access to their
data?

------
kazuki
Great post. I felt like $150/hour is way too cheap for his resume (10+ years
of experience, PhD, author of book, based in Seattle). I am pretty sure some
companies will be happy paying twice as much.

------
FpUser
"As a freelance AI engineer, you are expected to, for example, start with a
client, familiarize yourself with the product and the codebase, submit the
first PR within a couple of days"

Although I have my doubts maybe this is true specifically for AI area. In
general contracting, getting familiar with the large project and "submitting
the first PR within a couple of days" is a wet dream. I will exclude cases
when one is hired to find and fix bug in some simple, short piece of code.

~~~
sbacic
Are you trying to say it's unreasonable to expect a freelancer to start
contributing within a couple of days? Or the opposite?

Because I can't remember a single project where I wasn't expected to
contribute something immediately, especially with new clients (that need to be
convinced of the value).

~~~
FpUser
I guess we are talking about different kind of projects. I had few very short
limited scope projects (shortest one 3 days I think) but generally I design
and develop products from ground zero (some I own, some made for corporate
clients). It could take me a month sometime to write first line of code. And
no one in their right mind would expect me to start coding big product with no
design in place on a first day. As for trust - I have big portfolio and
stellar references from satisfied clients and some of my products can be seen
on the web. In my case this is normally what counts, not me banging on
keyboard from the first hour. You do not start coding major project without
without first having a clear and documented picture of what exactly is being
built. Doing it other way is a recipe for major disappointment.

------
angel_j
This person, like many, makes the error of commodity pricing for labor. They
charges less, the more the client buys. That's how corn is bought. Don't sell
your time like corn.

This kind of thinking/pricing has been inculcated by so much free-market
idealism, and of course on some level human labor can be thought of as a
commodity, but that's neo-evil.

~~~
namenotrequired
Would you mind sharing what you think are the practical downsides of this
pricing approach?

~~~
angel_j
I'm more interested from a labor perspective, but practically, you get less
money for your work, and no benefit. Especially as a freelancer, paying all
your own taxes, insurance, health, and company costs. That means you need an
addition 20% revenue, not less.

If you work in demanding field, and have a client that needs—and can
afford—your work, giving them them a discount only takes away from your stack.
What possible benefit is there? Repeat business? They will repeat if they need
it, not b/c it's on sale.

Assume you are literally putting money in their pocket (ROI) and get paid what
you're worth.

------
t0ughcritic
I think the lawyer example is great! People here don’t question how a lawyer
or good accountant can charge 500$/hr easily but an engineer, oh heavens no it
can’t be possible. Big business pay this because as others have said it’s
about returns, and they aren’t experts in the field and they know they’ll make
more back. As startup owners or employees, it may be hard to fathom.

If anything more engineers need to be paid even more. Heck even consider a
real estate agent that puts your house up for sale, and your house is worth 1m
and they get 2.5%, let’s say they walk away with 15k how many hours do you
think they spent marketing and managing the sale? In a hot market, where
prices are higher and things move fast, even they are reeling in 500$/hr so a
PhD level engineer pulling that shouldn’t shock you.

------
polm23
Hey, quick clarification - I posted this because I thought it was interesting,
but I am not the author and can't answer questions about it.

------
JoeMayoBot
Really nice article. I've been an independent consultant (freelancer) for
several years and there are so many parts of this that match my experience
too: whether freelancing is right for you, being prepared for uneven work
cycles, and what is fair game for billable hours. To me breadth of experience,
time flexibility, and just being independent are the strongest perks.

------
bigmanwalter
Congrats on making the leap! Do you find that your pipeline is busy enough are
are you still looking to grow it?

------
justingreet
Just wanted to say that even though you said you're not a native English
speaker, I never would have known if you hadn't mentioned it. Very well-
written and clearly organized post.

------
th_wacc_nt1
Great post! Anybody else here who is currently employed in this field and
wants to start a consulting agency or similar? Feel free to reach out at
ideavalid@icloud.com

------
bluekite2000
Who are your typical clients/industries? And how long is a typical engagement?
And why per hour vs per project pricing?

------
RocketSyntax
what are most people asking for - pytorch? or are they leaving those decisions
up to you?

~~~
m0zg
As a practitioner in the field and in similar circumstances, the answer will
probably be: it doesn't really matter. Personally I prefer PyTorch, but I can
work with whatever the client already uses, if anything. For one of my clients
I worked in both because there was no viable deployment option out of PyTorch
to one of their platforms. There still isn't, so both models are still
maintained and trained.

If they aren't 100% set in their ways, I do make them aware that things will
move at about half the speed with TF, so they'll effectively be paying twice
as much. If they are set in their ways, I do not mention it, since I'm not
going to change their mind anyway.

That said PyTorch 1.5.0 is just broken pretty much - tensor permute (which in
computer vision you end up doing for every input tensor) is 10x slower than it
used to be. There's an issue in GitHub already.

I'm beginning to worry about PyTorch.

~~~
m3at
Thanks for sharing your experience!

I'm working with TF and Pytorch as well, but so far for the later I have found
the project to be reasonably reliable (though I did find Chainer considerably
more polished). Can you share more about what worries you with Pytorch?

~~~
m0zg
1.5.0 is basically broken for computer vision. Input tensors are usually in
NHWC memory format, but PyTorch (and CUDA) prefers NCHW (planar). So you'd
normally run permute() to move things around. But
[https://github.com/pytorch/pytorch/issues/37142](https://github.com/pytorch/pytorch/issues/37142)
(and possibly other bugs) kinda gets in the way of that. I had to roll back,
since training got way slower than it was before, and it wasn't super fast to
begin with, even on my quad-GPU workstation.

The fact that such obvious, severe bugs make it through the release process
likely means that there isn't really much of a release process. And what's in
place doesn't even test the release on totally bread-and-butter models like
resnet50.

~~~
smhx
PyTorch maintainer here: we're looking into that and if needed will issue
1.5.1 asap. It didn't show up in release testing, which among other things
does end-to-end imagenet runs with ResNet50 and a few other models (i.e. time
and memory didn't regress). Will also figure out how to catch this early.

~~~
smhx
@m0zg if you could comment on the issue you quoted with any details, it would
be really helpful to us. Unlike what is reported in the github issue,
`permute` isn't the regression.

For reference, one of the core devs added more details based on where we are
with our investigation:
[https://github.com/pytorch/pytorch/issues/37142#issuecomment...](https://github.com/pytorch/pytorch/issues/37142#issuecomment-623205729)

------
Advaith
This is super cool. I'm curious if OP engages in cold emails to scout for
leads.

------
hoerzu
What are your clients?

~~~
sbmthakur
His clients are in education sector spanning from the US to Japan.

------
quaquaqua1
To anyone who is seriously considering paying $250 an hour for someone who
isn't guaranteeing they will deliver anything--

I will personally quit my job and sign a contract with you that clearly
defines what will be delivered by what date, if you are willing to pay flat
rate of $X.

You will not pay until after you accept delivery of the system.

~~~
TrackerFF
Just wait until you discover what the government is paying McKinsey (and the
likes) for their services. Hourly rates starting at $500, growing up into the
thousands.

