user16315: I'm the type of guy who likes to produce beauty from code - I mostly experiment with shaders.
Up to this day, I've only been doing so for the sake of games, but an idea popped up in my mind that I could decorate an actual film footage using handwritten postprocessing.
Is there any software that, apart from facilitating basic video editing tasks, lets me code (and later apply) my own effect completely from scratch, that is, lets me manipulate each input frame pixel by pixel? Either in some shader language or maybe CUDA or even C++?

user6330: You might want to look into using FFmpeg - it seems like the kind of tool you're looking for. Adobe's After Effects allows some custom scripting, but not nearly to the extent you're looking for.

user26831: You can code custom plugins for Shotcut video editor using MLT, frei0r, or WebVfx plugin. MLT and frei0r plugins require C or C++ coding; whereas WebVfx lets you use HTML5, CSS3, JavaScript, SVG, and WebGL. : https://shotcut.org/notes/make-plugins/
I like unity3d for audio/video things
Obviously, have a go on Milkdrop if you want fluid dynamics code, else see if you can splice a video streaming code into the open source MD2 codebase, everyone would be well chuffed. 
Research "VJ" programs, like dj programs for video. 

user1871: To add upon John's answer, one way to do this using FFmpeg would be to add your filters (in C), but another, more modular, way is to code a Frei0r plugin, which can then be used with various host applications, whether FFmpeg, MLT..etc

Frei0r is a minimalistic plugin API for video effects. The main
  emphasis is on simplicity for an API that will round up the most
  common video effects into simple filters, sources and mixers that can
  be controlled by parameters. It’s our hope that this way these simple
  effects can be shared between many applications, avoiding their
  reimplementation by different projects.


user456: As an alternative to Processing, if you prefer to use C#, you can write plugins for VVVV using their C# APIs. Also if you prefer to just write pixel and vertex shaders which you can apply to your audio/video, they support that as well.

user3592: You should check out Processing dot org.

user1950: I second the vote for Processing, but if C++ is more your thing than Java you could also look at openFrameworks. It allows for more processor-intensive computation and as such is good at real-time synthesis and manipulation of video.
Another tool worth checking out is NodeBox. It allows for coding, but wraps it all in a node-based GUI, the aim being to simplify the boring stuff.
Final Cut Pro ≤ 7 used to have a built in language called fxcript (based on c, but weird and buggy) that gave you pixel level access to the video stream and integrated into the editor seamlessly. I wrote a few plugins for that back in the day, but I jumped ship after the initial FCPX fiasco so I can't comment on the new language for visual effects FXPlug. I believe it's quite powerful, and, like processing, leverages openGL to do the heavy lifting.

16315: <p>I'm the type of guy who likes to produce beauty from code - I mostly experiment with shaders.</p>

<p>Up to this day, I've only been doing so for the sake of games, but an idea popped up in my mind that I could decorate an actual film footage using handwritten postprocessing.</p>

<p>Is there any software that, apart from facilitating basic video editing tasks, lets me code (and later apply) my own effect completely from scratch, that is, lets me manipulate each input frame pixel by pixel? Either in some shader language or maybe CUDA or even C++?</p>
 6330: <p>You might want to look into using <a href="https://ffmpeg.org/" rel="nofollow">FFmpeg</a> - it seems like the kind of tool you're looking for. Adobe's After Effects allows some <a href="http://cgi.tutsplus.com/tutorials/introduction-to-the-basics-of-after-effects-scripting--ae-22518" rel="nofollow">custom scripting</a>, but not nearly to the extent you're looking for.</p>
 26831: <p>You can code custom plugins for <em>Shotcut</em> video editor using MLT, frei0r, or WebVfx plugin. MLT and frei0r plugins require C or C++ coding; whereas WebVfx lets you use HTML5, CSS3, JavaScript, SVG, and WebGL. : <a href="https://shotcut.org/notes/make-plugins/" rel="nofollow noreferrer">https://shotcut.org/notes/make-plugins/</a></p>

<p>I like unity3d for audio/video things</p>

<p>Obviously, have a go on Milkdrop if you want fluid dynamics code, else see if you can splice a video streaming code into the open source MD2 codebase, everyone would be well chuffed. </p>

<p>Research "VJ" programs, like dj programs for video. </p>
 1871: <p>To add upon John's answer, one way to do this using FFmpeg would be to add your filters (in C), but another, more modular, way is to code a <a href="https://www.dyne.org/software/frei0r/" rel="nofollow">Frei0r</a> plugin, which can then be used with various host applications, whether FFmpeg, MLT..etc</p>

<blockquote>
  <p>Frei0r is a minimalistic plugin API for video effects. The main
  emphasis is on simplicity for an API that will round up the most
  common video effects into simple filters, sources and mixers that can
  be controlled by parameters. It’s our hope that this way these simple
  effects can be shared between many applications, avoiding their
  reimplementation by different projects.</p>
</blockquote>
 456: <p>As an alternative to Processing, if you prefer to use C#, you can write plugins for <a href="https://vvvv.org/" rel="nofollow">VVVV</a> using their <a href="https://vvvv.org/documentation/plugins" rel="nofollow">C# APIs</a>. Also if you prefer to just write pixel and vertex shaders which you can apply to your audio/video, <a href="https://vvvv.org/documentation/tutorial-of-effects-and-shaders" rel="nofollow">they support that as well</a>.</p>
 3592: <p>You should check out <a href="https://processing.org" rel="nofollow">Processing dot org</a>.</p>
 1950: <p>I second the vote for Processing, but if C++ is more your thing than Java you could also look at <a href="http://openframeworks.cc/" rel="nofollow">openFrameworks</a>. It allows for more processor-intensive computation and as such is good at real-time synthesis and manipulation of video.</p>

<p>Another tool worth checking out is <a href="https://www.nodebox.net/" rel="nofollow">NodeBox</a>. It allows for coding, but wraps it all in a node-based GUI, the aim being to simplify the boring stuff.</p>

<p>Final Cut Pro ≤ 7 used to have a built in language called <a href="http://www.joemaller.com/fcp/fxscript_reference.shtml" rel="nofollow">fxcript</a> (based on c, but weird and buggy) that gave you pixel level access to the video stream and integrated into the editor seamlessly. I wrote a few plugins for that <a href="http://web.archive.org/web/20091023111748/http://www.pureandapplied.com.au/plugins.html" rel="nofollow">back in the day</a>, but I jumped ship after the initial FCPX fiasco so I can't comment on the new language for visual effects <a href="https://developer.apple.com/library/mac/documentation/AppleApplications/Conceptual/FXPlug_overview/FXPlugSDKOverview/FXPlugSDKOverview.html" rel="nofollow">FXPlug</a>. I believe it's quite powerful, and, like processing, leverages openGL to do the heavy lifting.</p>
