Q:

How to modifiy the system apps running on my phone

I want to modify the system apps running on my specific phone. I am stumped about some important steps on how to do that - see my questions further below.
N.B.: I am not interested in deploying these modifications to other phones. Instead I want to play with the original code and improve it. Hence "create your own system app" is no sufficient answer.
Background
I am just starting to get into Android development (have sufficient Java knowledge, though). There are plenty of great tutorials out there (e.g. here and here) but they all seem to cover user application development. I'm more interested in getting (a little) "closer to the metal". Unfortunately I am unable to find good sources teaching how to do that.
Example
As an example let's say I want to modify how the built-in brightness control works (e.g. changing the minimum) or want to add features to the call screen (e.g. "Send SMS instead" when the line is busy).
Specific Questions
I obviously need these things to get started...
Sources
The sources for (parts of?) the OS running on my phone. This is the part that confuses me the most. Can I just download the official sources or is it likely that my vendor made modifications?
In my specific scenario (I use a Fairphone), could those modifications be contained in any of these sources:

some core apps deployed by the vendor
the image binaries (some way down the page)
sources for the Linux kernel and some other libraries

Tools
I really hope to stay in Java-land. But either case I need a development (I hope to use Android Studio or Eclipse ADT) and a debugging environment (I hope the Android Virtual Devices suffice).
Is it a valid assumption that I can use those tools for system apps or are they usually implemented in C?
Process
Assuming I can download the official sources and get started with, e.g. Android Studio. As opposed to developing a user app, do I have to do anything special to emulate the modified system app on AVD?
I'm asking because I assume that AVD already runs an unmodified version of that very app.

A:

I am just starting to get into Android development (have sufficient Java knowledge, though). ... I'm more interested in getting (a little) "closer to the metal".

You can of course follow your own bliss, but you may find it pretty frustrating to learn Android development by hacking on the OS itself. You'll find yourself doing a lot of OS debugging and working with harder-to-use tools than normal.

Example
As an example let's say I want to modify how the built-in brightness control works (e.g. changing the minimum)

I realize this is just an example, but this could be a tough one, because there are a lot of things that contribute to brightness control -- there's the app with the UI that sets it, but that talks to the framework and ultimately the driver for the hardware that actually deals with brightness curves for the display, in coordination with the ambient light sensor.

or want to add features to the call screen (e.g. "Send SMS instead" when the line is busy).

There's a dialer app, but implementing a feature like that will be...difficult, provided it's even feasible. I don't know offhand whether that sort of call state is available to the API.

Specific Questions
I obviously need these things to get started...
Sources
The sources for (parts of?) the OS running on my phone. This is the part that confuses me the most. Can I just download the official sources or is it likely that my vendor made modifications?

If you're replacing system apps, you'll need to run an OS signed with debug keys; you don't have access to the signing keys for the installed system apps on a retail build of the OS that would enable you to replace them. That will mean either getting a debug-keys installation or building it yourself from source, and flashing it to your phone. I would most strongly discourage you from using your daily phone for this sort of work -- you don't want it to be bricked when you need to receive an important call.
If you're not running a Nexus phone, then yes, your vendor has made modifications. The system apps are probably modified, and you almost certainly won't have access to the closed source of those apps. Also, there will be closed-source device drivers that are essential to the operation of the OS that you won't have access to. If you're trying to get AOSP Android up and running on such a device, as an individual hacker it's at best very, very difficult. If you're willing to use something like CyanogenMod, you could look to see if (and to what extent) they support your hardware.
If you are running a Nexus phone, then it's a goal of the AOSP project to try to have the OS bootable on select Nexus devices, though you won't necessarily enjoy the fullest functionality of the retail OS build, depending on the state of drivers -- even Nexus phones can have closed-source drivers and bits that make life more, um, exciting for building with AOSP.

In my specific scenario (I use a Fairphone), could those modifications be contained in any of these sources:
some core apps deployed by the vendor

Almost certainly

the image binaries (some way down the page)

Not sure what you mean by that

sources for the Linux kernel and some other libraries

It's unlikely there are significant modifications to the kernel itself, though vendors sometimes do...interesting...things with system configuration. I already mentioned device drivers. Vendors also have custom implementations/modifications of some framework APIs.

Tools

Vendor-specific tools are rare.

I really hope to stay in Java-land. But either case I need a development (I hope to use Android Studio or Eclipse ADT) and a debugging environment (I hope the Android Virtual Devices suffice).

Much of what you want to do can probably be implemented in Java, though it depends on how close you're getting to the hardware which can be difficult to know.

Is it a valid assumption that I can use those tools for system apps or are they usually implemented in C?

The "system apps" are implemented in Java, but again, you sort of have to know where the dividing line is.

Process
Assuming I can download the official sources and get started with, e.g. Android Studio. As opposed to developing a user app, do I have to do anything special to emulate the modified system app on AVD?

That's actually different, because you're dealing with an emulator system image instead of a phone image. The emulator system images are completely open source, and you can build them from AOSP.

I'm asking because I assume that AVD already runs an unmodified version of that very app.

That's correct.
To come back to my initial point, if you read between the lines of what I'm saying, I won't tell you it's impossible, but the difficulty ranges from freakin' tough on the easy end to spend-years-of-your-life on the hard end. If you really want to proceed, your best bet may be to buy the phone that's best supported by CyanogenMod and work that angle; there's a more active developer community there to help you. I know you're more excited about trying to customize the OS than you are in making a user-space app, but you'll experience less frustration with the latter than the former.

