Q:

How to transform function to lambda function

I have function:
FARPROC PUDSClass::getFunction(LPSTR szName)
{
    if(handleDLL == NULL)
    {
        return NULL;
    }
    return GetProcAddress(handleDLL, szName);
}

I am trying to rewrite it as lambda function.
FARPROC lambda_decToStrHex = [&](LPSTR szName)
{
    if (handleDLL == NULL)
    {
        return NULL;
    }
    return GetProcAddress(handleDLL, szName);
};

But I get error of inconsistent types "int" and "int (attribute(stdcall)*)()".
If I write it like this, it works ok:
auto getFunction = [&](LPSTR szName)
{
    return GetProcAddress(handleDLL, szName);
};

From what I understood, lambda cannot handle return NULL or GetProcAddress(), but why can a normal function do so?

A:

FARPROC lambda_decToStrHex = ...;

This will create a variable lambda_decToStrHex, it's type is FARPROC, which, according to documentation, is defined as
int (FAR WINAPI * FARPROC) ()

It is a function returning int and accepting no parameters. This simply doesn't match the type of your lambda.
If at all, you need a function pointer of correct type the lambda can decay to, a function accepting a C string and returning a FARPROC pointer (in short: pointer to function returning function pointer):
FARPROC (*lambda_decToStrHex)(LPSTR) = ...;

Solely: Your lambda cannot decay to a function pointer, as it has a non-empty closure:
... = [&](LPSTR name) { /* ... */ };
//     ^                    ^ uses dllHandle, which thus gets part of closure

So all that remains is assigning the lambda to a variable thetype of your lambda, which you get by auto.

Edit: return NULL;:
'Lambda' is a red herring here, it is not an issue of lambdas, but of any function having deduced return type (auto f() { ... }). NULL typically is defined as (void*)0, (but can look differently as well), so you get two different return types (void* and FARPROC) at the two exit points of the (in this case) lambda. Which return type should now be deduced?
At very first, you shouldn't be using NULL at all any more, it is a macro coming from C, whereas C++ provides a keyword for: nullptr. Solely, it won't solve the issue, as it has it's own type (std::nullptr_t) which still differs from FARPROC. A cast then can solve the issue (that actually would work with the macro as well):
return static_cast<FARPROC>(nullptr);

Now both exit points have the same return type and we are fine. A trailing return type achieves the same:
[&](LPSTR) -> FARPROC { ... }

nullptr automatically decays to correct pointer type (so you don't need a cast); NULL macro, without a cast and depending on how being defined, can (but needn't) fail, though.
Another solution would be one single, well defined exit point:
[&](LPSTR)
{
    FARPROC function = nullptr;
    if(dllHandle)
        function = get();
    return function;
}

Why did it work with the normal function then? Well, you explicitly specified the return type (FARPROC f(...)), which is equivalent to having a trailing return type. Additionally, the fact that it compiled and the error message:
"int" and "int (attribute(stdcall)*)()"
 ^^^

reveal that your compiler obviously defines NULL without the void* cast:
#define NULL 0

or similar – and 0 literal decays to null pointer as well...

