<commit_before>time: "Jun 8, 2020"
subject: "Papers with code and Preparing Applications for IPv6"
draft: false

description: |
  This week, I want to share a quick shell trick to check if a port open or not without using any external tool like `telnet` or `nc`.

  And it's a one liner: `echo > /dev/tcp/host/port`.  Example: `echo > /dev/tcp/127.0.0.1/1605/`. If the port is open, no outout, otherwise it will say:

  `-bash: connect: Connection refused
  -bash: /dev/tcp/127.0.0.1/1605: Connection refused`

  It only works on Linux though, so if you are on Mac, don't bother to try it.

links:
  - url: https://paperswithcode.com/
    title: "Papers with code"
    description: |
      Have ever feel frustrated reading a whitepaper without code? So either you have to re-implement to verify or use the research theory? This site is a directory of paper with its code,
      as in they have a github repository.

  - url: https://www.arin.net/resources/guide/ipv6/preparing_apps_for_v6.pdf
    title: "Preparing Applications for IPv6 (PDF)"
    description: |
      A software developers guide to writing and migrating networked applications for use on ipv6 networks. Discuss strategies for apps supporting both ipv4 and ipv6 during the transition.
      Not just in term of network, it also discussed how to store ipv6 into database effectively. How to deal with ipv6 in your programing language etc. So a great guide both in term of networking and programming.

  - url: https://nullprogram.com/blog/2016/08/03/
    title: "Appending to a File from Multiple Processes"
    description: "Is writing to a file from multiple processes safe? As in they won't corrupted the file or overwride each other? and will all get append properly."

  - url: https://dgraph.io/blog/post/solving-jepsen-with-opencensus/
    title: "Solving Jepsen with OpenCensus Distributed Tracing: A personal journey"
    description: |
      Jepsen is a framework to test distrubuted system. It uncovered many bugs in system like MongoDB, Riak, Redis, ElasticSearch.
      But what do you do if you are a developer and your app failed at Jepsen test. This is a journey of a dgraph developer try to fix
      one of the most complicated Jepsen test using distributed tracing. The test is so hard that you cannot just think of theory to quickly try fixing out blindly.

  - url: https://aaronparecki.com/2020/05/31/30/the-real-cause-of-the-sign-in-with-apple-zero-day
    title: "The Real Cause of the Sign In with Apple Zero-Day"
    description: |
      Recently Apple has a Zero-Day in thir Sign In with Apple system, which allow a way to login as anybody. The core issue is not validating user input data.
      If a single thing you can learn from this article is this: always validate user input data, especially in a multi steps flow, validate at every steps.

  - url: https://www.thedreaming.org/2020/05/26/avoid-csrf-attacks-with-api-design/
    title: "Avoiding CSRF Attacks with API Design"
    description: |
      All code examples are JavaScript/Node but generally all good rule: never use GET to modify state, csrf token, check origin header, Use SameSite cookies, write nagtive test cases.

  - url: https://www.percona.com/blog/2020/05/29/removing-postgresql-bottlenecks-caused-by-high-traffic/
    title: "Removing PostgreSQL Bottlenecks Caused by High Traffic"
    description: |
      There are many ways of addressing performance bottlenecks, but for the sake of discussion here are several ways we can look at this issue: Tuning Performance Parameters, Session Connections, Bloat, Autovacuum, Data Hotspots, Competing Application Processes

  - url: https://habr.com/en/company/postgrespro/blog/504498/
    title: "Locks in PostgreSQL"
    description: |
      A series about Postgres locks. This part is about dead lock, object-level locks and predicate locks.
  - url: https://ketancmaheshwari.github.io/posts/2020/05/24/SMC18-Data-Challenge-4.html
    title: "Running Awk in parallel to process 256M records"
    description: |
      Awk crunches massive data; a High Performance Computing (HPC) script calls hundreds of Awk concurrently. Fast and scalable in-memory solution on a fat machine.

  - url: http://dranger.com/ffmpeg/ffmpeg.html
    title: "How to Write a Video Player in Less Than 1000 Lines SD"
    description: "Leverage ffmpeg. ffmpeg is awesome but it's huge so knowing how to put thing together is a nice first step to get into video programming."

  - url: https://zserge.com/posts/jvm/
    title: "How to Write a (Toy) JVM"
    description: |
      The code example is in Go but it introduce you to structure of JVM and how much you can achieve with a short amount of code. Also a good chance to understand some aspect of JVM, regardless you like Java or not, I think this is a great artcile and will level up your knowledge.

#  - url: https://github.com/EmilHernvall/dnsguide
#    title: "A guide to writing a DNS Server from scratch in Rust"
#    description: |
#      Writing DNS server is like hello world for networking to me. DNS is small enough, not massive like HTTP, also relying on UDP, so I think it's a good exercise to learn UDP, DNS protocol.
#      This artcile is focus on Rust, and I'm not a Rust programmer, only toy with Rust and I was able to follow it.
#  - url: https://medium.com/@Zack.Grannan/the-epic-migration-of-spotify-to-google-cloud-platform-92372ae2d552
#    title: "Spotify’s Cloud Migration: 100+ Teams, 1200 Services, 20k Data Jobs"
#
# - url: https://fzakaria.com/2020/05/31/containers-from-first-principles.html
#
#  - url: https://blog.twitter.com/engineering/en_us/topics/open-source/2020/hunting-a-linux-kernel-bug.html
#    title: "Hunting a Linux kernel bug"
#
#  - url: https://csswizardry.com/2020/05/the-fastest-google-fonts/
#    title: "The Fastest Google Fonts"
#
#    https://www.gutenberg.org/files/30688/30688-h/30688-h.htm
#
#    https://thorsten-hans.com/5-types-of-zsh-aliases#operating-system-specific-aliases
# https://github.com/LisaDziuba/Marketing-for-Engineers
#https://pragmaticwebsecurity.com/articles/oauthoidc/from-implicit-to-pkce.html

video:
  - url: https://www.youtube.com/watch?v=dhoP-dyIr54
    title: "What is a Secure Programming Language?"

  - url: https://www.youtube.com/watch?v=E-ZbrtoSuzw&feature=youtu.be
    title: "Going mouseless with Vim/Tmux/Hotkeys"

read_code:
  - url: https://github.com/carlini/printf-tac-toe
    title: printf-tac-toe
    description: "tic-tac-toe in a single call to printf"
    category: ["C"]

  - url: https://github.com/lihaoyi/Metascala
    title: Metascala
    description: |
      A JVM written in 3000 line of Scala that can compile itself
    category: ['Scala']

  - url: https://github.com/cowbell/sharedrop
    title: sharedrop
    description: |
      Easy P2P file transfer powered by WebRTC - inspired by Apple AirDrop 
    category: ['JavaScript', 'WebRTC']

  - url: https://github.com/niieani/hashids.js/blob/master/lib/hashids.ts
    title: hashids
    description: |
      Ever wonder how to generate YouTube-like ids from numbers? Here you go, in 400 lines of TypeScript
    category: ['TypeScript']

  - url: github.com/kurolabs/airshare
    title: "Airshare"
    description: |
      Cross-platform content sharing in a local network
    category: ['Python']

  - url: https://github.com/miguelmota/streamhut
    title: streamhut
    description: "stream your terminal. Checkout demo on [streamhut](https://streamhut.io)"
    category: ["Go"]

  - url: https://github.com/free5gc/free5gc
    title: free5gc
    description: |
      Open source 5G core network base on 3GPP R1
    category: ['Go']

  - url: https://github.com/ankane/pgslice
    title: pgslice
    description: |
      Postgres partitioning as easy as pie
    category: ['Ruby']

  - url: https://github.com/theryangeary/choose
    title: choose
    description: |
      A human-friendly and fast alternative to awk and cut
    category: ['Rust']

tool:
  - url: https://github.com/electerious/Ackee
    title: Ackee
    description: |
      Self-hosted, Node.js based analytics tool for those who care about privacy. Ackee runs on your own server, analyses the traffic of your websites and provides useful statistics in a minimal interface

  - url: https://github.com/ankane/blazer
    title: blazer
    description: |
      Explore your data with SQL. Easily create charts and dashboards, and share them with your team

  - url: https://smocker.dev/
    title: smocker
    description: |
      A simple and efficient HTTP mock server

  - url: https://github.com/sqlancer/sqlancer
    title: sqlancer
    description: "Detecting Logic Bugs in DBMS. It discovered 179 bugs in SQLite"

  #https://github.com/jay0lee/got-your-back

  - url: https://github.com/sickcodes/Docker-OSX
    title: "Mac in Docker!"
    description: "Run near native OSX-KVM in Docker! X11 Forwarding!"

  - url: https://as-ideas.github.io/TransformerTTS/
    title: TransformerTTS
    description: "A Text-to-Speech Transformer in TensorFlow 2"

  - url: https://github.com/milesmcc/shynet
    title: shynet
    description: |
      Modern, privacy-friendly, and detailed web analytics that works without cookies or JS

  - url: https://github.com/zalando/nakadi
    title: nakadi
    description: |
       a distributed event bus broker that implements a RESTful API abstraction on top of Kafka-like queues, which can be used to send, receive, and analyze streaming data in real time, in a reliable and highly available manner.

  - url: https://puree.cc/
    title: puree
    description: "Password-based Uniform-Random-Equivalent Encryption"
<commit_msg>clean up<commit_after>time: "Jun 8, 2020"
subject: "Papers with code and Preparing Applications for IPv6"
draft: false

description: |
  This week, I want to share a quick shell trick to check if a port open or not without using any external tool like `telnet` or `nc`.

  And it's a one liner: `echo > /dev/tcp/host/port`.  Example: `echo > /dev/tcp/127.0.0.1/1605/`. If the port is open, no outout, otherwise it will say:

  `-bash: connect: Connection refused
  -bash: /dev/tcp/127.0.0.1/1605: Connection refused`

  It only works on Linux though, so if you are on Mac, don't bother to try it.

links:
  - url: https://paperswithcode.com/
    title: "Papers with code"
    description: |
      Have ever feel frustrated reading a whitepaper without code? So either you have to re-implement to verify or use the research theory? This site is a directory of paper with its code,
      as in they have a github repository.

  - url: https://www.arin.net/resources/guide/ipv6/preparing_apps_for_v6.pdf
    title: "Preparing Applications for IPv6 (PDF)"
    description: |
      A software developers guide to writing and migrating networked applications for use on ipv6 networks. Discuss strategies for apps supporting both ipv4 and ipv6 during the transition.
      Not just in term of network, it also discussed how to store ipv6 into database effectively. How to deal with ipv6 in your programing language etc. So a great guide both in term of networking and programming.

  - url: https://nullprogram.com/blog/2016/08/03/
    title: "Appending to a File from Multiple Processes"
    description: "Is writing to a file from multiple processes safe? As in they won't corrupted the file or overwride each other? and will all get append properly."

  - url: https://dgraph.io/blog/post/solving-jepsen-with-opencensus/
    title: "Solving Jepsen with OpenCensus Distributed Tracing: A personal journey"
    description: |
      Jepsen is a framework to test distrubuted system. It uncovered many bugs in system like MongoDB, Riak, Redis, ElasticSearch.
      But what do you do if you are a developer and your app failed at Jepsen test. This is a journey of a dgraph developer try to fix
      one of the most complicated Jepsen test using distributed tracing. The test is so hard that you cannot just think of theory to quickly try fixing out blindly.

  - url: https://aaronparecki.com/2020/05/31/30/the-real-cause-of-the-sign-in-with-apple-zero-day
    title: "The Real Cause of the Sign In with Apple Zero-Day"
    description: |
      Recently Apple has a Zero-Day in thir Sign In with Apple system, which allow a way to login as anybody. The core issue is not validating user input data.
      If a single thing you can learn from this article is this: always validate user input data, especially in a multi steps flow, validate at every steps.

  - url: https://www.thedreaming.org/2020/05/26/avoid-csrf-attacks-with-api-design/
    title: "Avoiding CSRF Attacks with API Design"
    description: |
      All code examples are JavaScript/Node but generally all good rule: never use GET to modify state, csrf token, check origin header, Use SameSite cookies, write nagtive test cases.

  - url: https://www.percona.com/blog/2020/05/29/removing-postgresql-bottlenecks-caused-by-high-traffic/
    title: "Removing PostgreSQL Bottlenecks Caused by High Traffic"
    description: |
      There are many ways of addressing performance bottlenecks, but for the sake of discussion here are several ways we can look at this issue: Tuning Performance Parameters, Session Connections, Bloat, Autovacuum, Data Hotspots, Competing Application Processes

  - url: https://habr.com/en/company/postgrespro/blog/504498/
    title: "Locks in PostgreSQL"
    description: |
      A series about Postgres locks. This part is about dead lock, object-level locks and predicate locks.
  - url: https://ketancmaheshwari.github.io/posts/2020/05/24/SMC18-Data-Challenge-4.html
    title: "Running Awk in parallel to process 256M records"
    description: |
      Awk crunches massive data; a High Performance Computing (HPC) script calls hundreds of Awk concurrently. Fast and scalable in-memory solution on a fat machine.

  - url: http://dranger.com/ffmpeg/ffmpeg.html
    title: "How to Write a Video Player in Less Than 1000 Lines SD"
    description: "Leverage ffmpeg. ffmpeg is awesome but it's huge so knowing how to put thing together is a nice first step to get into video programming."

  - url: https://zserge.com/posts/jvm/
    title: "How to Write a (Toy) JVM"
    description: |
      The code example is in Go but it introduce you to structure of JVM and how much you can achieve with a short amount of code. Also a good chance to understand some aspect of JVM, regardless you like Java or not, I think this is a great artcile and will level up your knowledge.

video:
  - url: https://www.youtube.com/watch?v=dhoP-dyIr54
    title: "What is a Secure Programming Language?"

  - url: https://www.youtube.com/watch?v=E-ZbrtoSuzw&feature=youtu.be
    title: "Going mouseless with Vim/Tmux/Hotkeys"

read_code:
  - url: https://github.com/carlini/printf-tac-toe
    title: printf-tac-toe
    description: "tic-tac-toe in a single call to printf"
    category: ["C"]

  - url: https://github.com/lihaoyi/Metascala
    title: Metascala
    description: |
      A JVM written in 3000 line of Scala that can compile itself
    category: ['Scala']

  - url: https://github.com/cowbell/sharedrop
    title: sharedrop
    description: |
      Easy P2P file transfer powered by WebRTC - inspired by Apple AirDrop 
    category: ['JavaScript', 'WebRTC']

  - url: https://github.com/niieani/hashids.js/blob/master/lib/hashids.ts
    title: hashids
    description: |
      Ever wonder how to generate YouTube-like ids from numbers? Here you go, in 400 lines of TypeScript
    category: ['TypeScript']

  - url: github.com/kurolabs/airshare
    title: "Airshare"
    description: |
      Cross-platform content sharing in a local network
    category: ['Python']

  - url: https://github.com/miguelmota/streamhut
    title: streamhut
    description: "stream your terminal. Checkout demo on [streamhut](https://streamhut.io)"
    category: ["Go"]

  - url: https://github.com/free5gc/free5gc
    title: free5gc
    description: |
      Open source 5G core network base on 3GPP R1
    category: ['Go']

  - url: https://github.com/ankane/pgslice
    title: pgslice
    description: |
      Postgres partitioning as easy as pie
    category: ['Ruby']

  - url: https://github.com/theryangeary/choose
    title: choose
    description: |
      A human-friendly and fast alternative to awk and cut
    category: ['Rust']

tool:
  - url: https://github.com/electerious/Ackee
    title: Ackee
    description: |
      Self-hosted, Node.js based analytics tool for those who care about privacy. Ackee runs on your own server, analyses the traffic of your websites and provides useful statistics in a minimal interface

  - url: https://github.com/ankane/blazer
    title: blazer
    description: |
      Explore your data with SQL. Easily create charts and dashboards, and share them with your team

  - url: https://smocker.dev/
    title: smocker
    description: |
      A simple and efficient HTTP mock server

  - url: https://github.com/sqlancer/sqlancer
    title: sqlancer
    description: "Detecting Logic Bugs in DBMS. It discovered 179 bugs in SQLite"

  - url: https://github.com/sickcodes/Docker-OSX
    title: "Mac in Docker!"
    description: "Run near native OSX-KVM in Docker! X11 Forwarding!"

  - url: https://as-ideas.github.io/TransformerTTS/
    title: TransformerTTS
    description: "A Text-to-Speech Transformer in TensorFlow 2"

  - url: https://github.com/milesmcc/shynet
    title: shynet
    description: |
      Modern, privacy-friendly, and detailed web analytics that works without cookies or JS

  - url: https://github.com/zalando/nakadi
    title: nakadi
    description: |
       a distributed event bus broker that implements a RESTful API abstraction on top of Kafka-like queues, which can be used to send, receive, and analyze streaming data in real time, in a reliable and highly available manner.

  - url: https://puree.cc/
    title: puree
    description: "Password-based Uniform-Random-Equivalent Encryption"

{'fraction_non_alphanumeric': 0.08553473110505534, 'fraction_numerical': 0.015627606918413882, 'mean_word_length': 3.3277978339350183, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 7, 'https://': 70, 'lorem ipsum': 0, 'www.': 11, 'xml': 0}, 'pii_count': 4, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4082589', 'n_tokens_mistral': 5667, 'n_tokens_neox': 5289, 'n_words': 2111}