<commit_before>* Lambda Days 2015

** Day 1

*** <NAME> - Rediscovering a Useful Pattern Language

- Programming should be a joy for you!
- *Timeless way of building*, <NAME>
- Patterns are built by many, not by few.
  - Urban architecture, city buildings, common paths.
  - Emergent Architecture.
- Pattern Language.
  - Patterns are created with certain emotion, this is the pattern context.
  - Discovered, not prescribed.
  - Experimental, not theoretical.
  - Sample patterns:
    - *REPL*, Text based representation, Pattern matching.
- Emotions are signals, they point to something real.

*** <NAME> - Megacore, Megafast, Megacool

- Future: *manycores*, *megacores* not *multicores*.
  - It is happening right now: *exascale*, *big data*.
  - Linking multiple nodes into systems.
  - Heterogeneous structure.
    - Not shared memory, *NUMA*.
    - Approx. 34 petaflops, 3 millions cores.
- Not only PC - mobile and embedded systems as well e.g.:
  - Samsung Exynos - *8 cores*.
  - NVIDIA Tegra K1 - *4 ARM cores*, *192-core Kepler GPU*.
- Performance trade-offs related with energy usage.
  - *Dark Silicon*
  - Power cores on demand, if not used switch them off.
- Spawning 20k threads in 4.5s of execution.
  - *How to manage that?*
- Case study:
  - Building a wall - 1 worker, 4 workers
    - Not exactly a 4x speedup - *Amdahl's Law*.
    - Real dependencies, accidental dependencies.
- We need better abstractions, you shouldn't think in terms of threads.
  - No worrying on communication, dependencies, deadlocks...
    - Ill-defined parallel domains are constraining.
  - Purity means no side-effects.
- Models:
  - Concurrency - illusion, execution way.
    - Programming models based on concurrency are low-level.
  - Parallelism - Hardware artifact, real.
- Patterns of parallelism
  - Pipelines, Map-Reduce, Task farm, Divide & Conquer, parallel
    search, parallel completion...
  - Stateless computation can be redone, anywhere, anytime.
- Projects:
  - https://github.com/ParaPhrase/skel

*** <NAME> - Thinking like Erlanger

- Sample code: https://github.com/lehoff/egol
- *Feedback is a breakfast of champions*.
- Fail fast, learn quick.
- Languages - Generic vs. Specific:
  - Second group better fits the certain domains.
- Two books:
  - <NAME>, "Principles of Protocol Design" (Springer)
  - <NAME>, "Communicating Sequential Processes"
- Golden trinity of Erlang:
  - Fail fast.
  - Share nothing.
  - Failure handling.
- Generic way of handling:
  - Query about past state - keep history - *in-process*, ETS / DETS,
    snapshots.
  - Query about future state - queue the response and wait for it.
- Protocols and MSC are going hand by hand.
- Unique Erlang features which helps:
  - Links
  - Monitors
  - Supervisors
  - `gproc` for PUB / SUB.

*** <NAME> - Understanding Cancer Behavior with F#

- Using F# in data mining and machine learning.
- Why not:
  - Python?
  - Matlab?
  - R?
- All are slow on huge data sets, not enough expressive, and are using
  for fast prototyping. Moreover *R* is written for statisticians by
  statisticians.
- Why F#?
  - With complicated domain (predicting and classifying breast cancer)
    and algorithms (e.g. integrative clustering) code quality and
    additional power features are key of success. Expressiveness is
    also one of the most important things.
  - Details:
    - Types
    - Expressiveness
    - Pattern Matching
    - Legibility (`a |> b`) - Data Transformations
    - Type providers
      - Typed access to external data sources (CSV, SQL, XLS ...).
      - Example: CSV file and access to the `Rows` property and for
        each row full intellisense with column name and expected type
        of values in that column.
      - It builds it automatically, based on exemplary data or value
        representation. Example with sample JSON response passed as a
        initial thing to the Typed Provider. After using it, you can
        use auto-completion because it is resolved in compile time.
      - Not only limited to the documents - there is an e.g. RProvider
        which introduces whole (!) without exceptions standard library
        of R programming language.

*** <NAME> - A Year of Haskell

- 11 millions professional developers in whole world.
  - 1000 professional Haskell developers.
- Talk about transition from purely Ruby shop, to the Ruby company
  that uses Haskell with 5-6 full-time Haskell developers.
- *Freedom is another world for nothing to lose*.
- Projects are acquiring legacy status much more easily in dynamically
  typed languages.
- Most of web applications are I/O bound, not the CPU bound - which is
  not a main use case for Haskell. It is exactly the opposite, CPU
  intensive tasks are most popular ones in the Haskell community.
- Good things about Haskell:
  - Stable
  - Focused on core abstractions.
  - Active community.
  - Language evolve permanently.
- Getting better:
  - Tooling
  - More and better libraries.
- Bad:
  - It is a small community, which directly matches to the smaller adoption.
  - It is not popular in enterprise companies, because it is hard to
    find many, good devlopers.
- How to sell Haskell?
  - Sell focus on increased productivity, instead safety.
  - Not only for geniuses.
  - Selling is a part of our job.
  - Most clients are resisting new things or directly finishing the discussion.
    - *Permission Denied*.
  - Maintenance is an important thing, but most of very intelligent
    members of community are leaving Ruby.
  - Sell these values (it is verified by them):
    - Commitment
    - Scarcity
    - Consistency
    - Reciprocity
    - Linking and Authority

*** <NAME> - State, Communication and Side-Effects in Idris

- All demos are in: https://github.com/edwinb/idris-demos
- Dependent Typing
  - Type Driven Development ;)
  - Using proofs in code, formalizing assumptions in type system.
  - Treats type as a first-class value in a language.
- In other words: specify the types, and determine the execution or
  actions invoked further in the program flow based on a type of
  returned value.
- So you can define that client corresponding to the protocol sends
  some value to the server which is a certain type - in Idris you can
  verify, that all commands are invoked within the valid state and in
  certain order.
- You can define full Echo protocol inside the type system, change the
  specification, compile it and compiler checks compliance and report
  all invalid places, which are not spec compliant.
  - Imagine that compiler can infer the deadlocks, logic errors in
    protocol specifications or prevent from timing attacks and
    introduce a notion of time to the system, and verify that all
    time-related constraints are matched.

*** <NAME> - Building systems in Clojure

- Case study and explanation about *BirdWatch*.
  - Project which provides a real-time dashboard to the Twitter
    hashtags, with searching capabilities.
  - Technologies: ElasticSearch, Redis, Clojure, Twitter Streaming
    API.
  - It is a fun project on a side (first written in Scala, now in
    Clojure). Front-end is written in ClojureScript and Reagent
    (previously Om).
- Matthias is doing transcripts for the popular talks from Clojure
  community in the Github.
- Also, he is writing the book about building systems in Clojure, it
  is on LeanPub. It is also available on Github as a Markdown source.
- Very nice libraries:
  - Carmine - Redis library for Clojure.
    - Ideal case study - Redis, Clojure and this library.
  - Components from Stuart Sierra.
    - Very nice structure for building application from scratch.
- Nice analogies:
  - Channels are like conveyor belts, put value on it and you
    shouldn't care about wiring and if someone consume it, somewhere.
- Matthias is also author of tool which visualizes messages flowing
  through system - https://github.com/matthiasn/inspect

** Day 2

*** <NAME> - How computers change our brains. Can screen-time cause mental or neurological disorders?

- Brain as a most important organ, also the irreplaceable one (state
  of today's medicine).
- Fact: receives 25% of total body glucose utilization, 20% of oxygen
  utilization and so on.
- Brain disorders will be the most important cause of health injuries
  soon (WHO).
- Fact: ~800 billions EUR spent on brain disorders in 2010 in Europe
  only.
- Neuroplasticity - ability to create new nerve cells are produced in
  adulthood.
- Animals brains are not as plastic as the human one.
- *Neurons that fire together are wired together*.
- How computers and *programmer's lifestyle* affects a brain?
  - Disorders in melatonin hormone levels.
  - Huge amount of lights (e.g. screen light) before sleep will
    increase the probability of having insomnia or other sleep
    disorders.
  - Gray matter atrophy in *internet / gaming addiction*.
    - Similar changes are caused by mental disorders e.g.:
      - Losing focus, Shorter attention span ...
      - Less analytic capabilities.
  - Neural synchrony is crucial to memory formation.
    - Seeing and hearing - synergy is the key.
  - Tools are affecting your learning process, attention span.
    - Before Google you have to know where to look at!
    - Association is a key for memorizing, and using some kind of
      tooling will decrease that abilities.
- *Not used or less used parts of brain cannot be allocated in the
  different area*.
- Environmental factors + particular set of Genes = Schizophrenia
  (researches).

*** <NAME> - You say big data, I say fast data

- Big data problem - it is a +100 TB data storage, only a few
  companies in the world.
- Problem space is wider, but amount is smaller:
  - Huge volume data streams, with which we have to perform analysis.
  - Streaming Data and Analysis of it is crucial from business
    perspective.
- Case Studies:
  - Recommendation engines
  - Analytics
- *Hurdles*:
  - Typical issues with Lambda Architectures.
    - Duplication logic, parallel stacks, multiple operation teams...
    - Latency, temporal queries...
      - Probabilistic Data Structures.
      - Solution - proper store e.g. EventStore, BlinkDB.
    - Duplication and Cache Utilization.
      - Tachyon - shared layer, memory between nodes.
- Apache Spark as a solution for streaming data and capturing ~80% of
  requirements pushed to tools which are inappropriately used for this
  cases (e.g. Hadoop).

*** <NAME> - Implementing graph representation model for parallel and distributed systems using Erlang

-

*** <NAME> - An invitation to functional programming
*** <NAME> - Dissecting the Rabbit: RabbitMQ Internal Architecture
*** <NAME> - Functional Survival In The World Of Concurrent I/O
*** <NAME> - Lessons from Erlang VM
*** <NAME> - Modular program syntax and semantics in Scala
<commit_msg>LambdaDays 2015 - Adding notes after talk about Graphs processing in Erlang.<commit_after>* Lambda Days 2015

** Day 1

*** <NAME> - Rediscovering a Useful Pattern Language

- Programming should be a joy for you!
- *Timeless way of building*, <NAME>
- Patterns are built by many, not by few.
  - Urban architecture, city buildings, common paths.
  - Emergent Architecture.
- Pattern Language.
  - Patterns are created with certain emotion, this is the pattern context.
  - Discovered, not prescribed.
  - Experimental, not theoretical.
  - Sample patterns:
    - *REPL*, Text based representation, Pattern matching.
- Emotions are signals, they point to something real.

*** <NAME> - Megacore, Megafast, Megacool

- Future: *manycores*, *megacores* not *multicores*.
  - It is happening right now: *exascale*, *big data*.
  - Linking multiple nodes into systems.
  - Heterogeneous structure.
    - Not shared memory, *NUMA*.
    - Approx. 34 petaflops, 3 millions cores.
- Not only PC - mobile and embedded systems as well e.g.:
  - Samsung Exynos - *8 cores*.
  - NVIDIA Tegra K1 - *4 ARM cores*, *192-core Kepler GPU*.
- Performance trade-offs related with energy usage.
  - *Dark Silicon*
  - Power cores on demand, if not used switch them off.
- Spawning 20k threads in 4.5s of execution.
  - *How to manage that?*
- Case study:
  - Building a wall - 1 worker, 4 workers
    - Not exactly a 4x speedup - *Amdahl's Law*.
    - Real dependencies, accidental dependencies.
- We need better abstractions, you shouldn't think in terms of threads.
  - No worrying on communication, dependencies, deadlocks...
    - Ill-defined parallel domains are constraining.
  - Purity means no side-effects.
- Models:
  - Concurrency - illusion, execution way.
    - Programming models based on concurrency are low-level.
  - Parallelism - Hardware artifact, real.
- Patterns of parallelism
  - Pipelines, Map-Reduce, Task farm, Divide & Conquer, parallel
    search, parallel completion...
  - Stateless computation can be redone, anywhere, anytime.
- Projects:
  - https://github.com/ParaPhrase/skel

*** <NAME> - Thinking like Erlanger

- Sample code: https://github.com/lehoff/egol
- *Feedback is a breakfast of champions*.
- Fail fast, learn quick.
- Languages - Generic vs. Specific:
  - Second group better fits the certain domains.
- Two books:
  - <NAME>, "Principles of Protocol Design" (Springer)
  - <NAME>, "Communicating Sequential Processes"
- Golden trinity of Erlang:
  - Fail fast.
  - Share nothing.
  - Failure handling.
- Generic way of handling:
  - Query about past state - keep history - *in-process*, ETS / DETS,
    snapshots.
  - Query about future state - queue the response and wait for it.
- Protocols and MSC are going hand by hand.
- Unique Erlang features which helps:
  - Links
  - Monitors
  - Supervisors
  - `gproc` for PUB / SUB.

*** <NAME> - Understanding Cancer Behavior with F#

- Using F# in data mining and machine learning.
- Why not:
  - Python?
  - Matlab?
  - R?
- All are slow on huge data sets, not enough expressive, and are using
  for fast prototyping. Moreover *R* is written for statisticians by
  statisticians.
- Why F#?
  - With complicated domain (predicting and classifying breast cancer)
    and algorithms (e.g. integrative clustering) code quality and
    additional power features are key of success. Expressiveness is
    also one of the most important things.
  - Details:
    - Types
    - Expressiveness
    - Pattern Matching
    - Legibility (`a |> b`) - Data Transformations
    - Type providers
      - Typed access to external data sources (CSV, SQL, XLS ...).
      - Example: CSV file and access to the `Rows` property and for
        each row full intellisense with column name and expected type
        of values in that column.
      - It builds it automatically, based on exemplary data or value
        representation. Example with sample JSON response passed as a
        initial thing to the Typed Provider. After using it, you can
        use auto-completion because it is resolved in compile time.
      - Not only limited to the documents - there is an e.g. RProvider
        which introduces whole (!) without exceptions standard library
        of R programming language.

*** <NAME> - A Year of Haskell

- 11 millions professional developers in whole world.
  - 1000 professional Haskell developers.
- Talk about transition from purely Ruby shop, to the Ruby company
  that uses Haskell with 5-6 full-time Haskell developers.
- *Freedom is another world for nothing to lose*.
- Projects are acquiring legacy status much more easily in dynamically
  typed languages.
- Most of web applications are I/O bound, not the CPU bound - which is
  not a main use case for Haskell. It is exactly the opposite, CPU
  intensive tasks are most popular ones in the Haskell community.
- Good things about Haskell:
  - Stable
  - Focused on core abstractions.
  - Active community.
  - Language evolve permanently.
- Getting better:
  - Tooling
  - More and better libraries.
- Bad:
  - It is a small community, which directly matches to the smaller adoption.
  - It is not popular in enterprise companies, because it is hard to
    find many, good devlopers.
- How to sell Haskell?
  - Sell focus on increased productivity, instead safety.
  - Not only for geniuses.
  - Selling is a part of our job.
  - Most clients are resisting new things or directly finishing the discussion.
    - *Permission Denied*.
  - Maintenance is an important thing, but most of very intelligent
    members of community are leaving Ruby.
  - Sell these values (it is verified by them):
    - Commitment
    - Scarcity
    - Consistency
    - Reciprocity
    - Linking and Authority

*** <NAME> - State, Communication and Side-Effects in Idris

- All demos are in: https://github.com/edwinb/idris-demos
- Dependent Typing
  - Type Driven Development ;)
  - Using proofs in code, formalizing assumptions in type system.
  - Treats type as a first-class value in a language.
- In other words: specify the types, and determine the execution or
  actions invoked further in the program flow based on a type of
  returned value.
- So you can define that client corresponding to the protocol sends
  some value to the server which is a certain type - in Idris you can
  verify, that all commands are invoked within the valid state and in
  certain order.
- You can define full Echo protocol inside the type system, change the
  specification, compile it and compiler checks compliance and report
  all invalid places, which are not spec compliant.
  - Imagine that compiler can infer the deadlocks, logic errors in
    protocol specifications or prevent from timing attacks and
    introduce a notion of time to the system, and verify that all
    time-related constraints are matched.

*** <NAME> - Building systems in Clojure

- Case study and explanation about *BirdWatch*.
  - Project which provides a real-time dashboard to the Twitter
    hashtags, with searching capabilities.
  - Technologies: ElasticSearch, Redis, Clojure, Twitter Streaming
    API.
  - It is a fun project on a side (first written in Scala, now in
    Clojure). Front-end is written in ClojureScript and Reagent
    (previously Om).
- Matthias is doing transcripts for the popular talks from Clojure
  community in the Github.
- Also, he is writing the book about building systems in Clojure, it
  is on LeanPub. It is also available on Github as a Markdown source.
- Very nice libraries:
  - Carmine - Redis library for Clojure.
    - Ideal case study - Redis, Clojure and this library.
  - Components from Stuart Sierra.
    - Very nice structure for building application from scratch.
- Nice analogies:
  - Channels are like conveyor belts, put value on it and you
    shouldn't care about wiring and if someone consume it, somewhere.
- Matthias is also author of tool which visualizes messages flowing
  through system - https://github.com/matthiasn/inspect

** Day 2

*** <NAME> - How computers change our brains. Can screen-time cause mental or neurological disorders?

- Brain as a most important organ, also the irreplaceable one (state
  of today's medicine).
- Fact: receives 25% of total body glucose utilization, 20% of oxygen
  utilization and so on.
- Brain disorders will be the most important cause of health injuries
  soon (WHO).
- Fact: ~800 billions EUR spent on brain disorders in 2010 in Europe
  only.
- Neuroplasticity - ability to create new nerve cells are produced in
  adulthood.
- Animals brains are not as plastic as the human one.
- *Neurons that fire together are wired together*.
- How computers and *programmer's lifestyle* affects a brain?
  - Disorders in melatonin hormone levels.
  - Huge amount of lights (e.g. screen light) before sleep will
    increase the probability of having insomnia or other sleep
    disorders.
  - Gray matter atrophy in *internet / gaming addiction*.
    - Similar changes are caused by mental disorders e.g.:
      - Losing focus, Shorter attention span ...
      - Less analytic capabilities.
  - Neural synchrony is crucial to memory formation.
    - Seeing and hearing - synergy is the key.
  - Tools are affecting your learning process, attention span.
    - Before Google you have to know where to look at!
    - Association is a key for memorizing, and using some kind of
      tooling will decrease that abilities.
- *Not used or less used parts of brain cannot be allocated in the
  different area*.
- Environmental factors + particular set of Genes = Schizophrenia
  (researches).

*** <NAME> - You say big data, I say fast data

- Big data problem - it is a +100 TB data storage, only a few
  companies in the world.
- Problem space is wider, but amount is smaller:
  - Huge volume data streams, with which we have to perform analysis.
  - Streaming Data and Analysis of it is crucial from business
    perspective.
- Case Studies:
  - Recommendation engines
  - Analytics
- *Hurdles*:
  - Typical issues with Lambda Architectures.
    - Duplication logic, parallel stacks, multiple operation teams...
    - Latency, temporal queries...
      - Probabilistic Data Structures.
      - Solution - proper store e.g. EventStore, BlinkDB.
    - Duplication and Cache Utilization.
      - Tachyon - shared layer, memory between nodes.
- Apache Spark as a solution for streaming data and capturing ~80% of
  requirements pushed to tools which are inappropriately used for this
  cases (e.g. Hadoop).

*** <NAME> - Implementing graph representation model for parallel and distributed systems using Erlang

- Traditional models of representing systems are based on passive
  representations of graphs.
- Moreover - they are *synchronous* which leads to the ineffective
  resource usage.
  - You can do asynchronous model from a synchronous one.
- In Erlang we can use lightweight processes to simulate graph
  structure, instead of passive data structure we are using an
  adaptive mesh of processes representing graphs.
  - Grid computing, Beowulf clusters etc.
- Three levels of processes, which manages the state of each type of
  layer inside process local state, ETS and DETS tables.

*** <NAME> - An invitation to functional programming

-

*** <NAME> - Dissecting the Rabbit: RabbitMQ Internal Architecture
*** <NAME> - Functional Survival In The World Of Concurrent I/O
*** <NAME> - Lessons from Erlang VM
*** <NAME> - Modular program syntax and semantics in Scala

{'fraction_non_alphanumeric': 0.06671458735792264, 'fraction_numerical': 0.004312862213037423, 'mean_word_length': 3.163860830527497, 'pattern_counts': {'":': 0, '<': 39, '<?xml version=': 0, '>': 41, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3854393', 'n_tokens_mistral': 6169, 'n_tokens_neox': 5798, 'n_words': 2986}