<commit_before># Gopherbot Development Notes - Current

`DevNotes.md` - TODO items and design notes for future developmen

## 2.0 Goals
* Job scheduling (DONE)
* Pipelines (DONE)
* Remote plugin execution over ssh (ON HOLD)
* GopherCI for CI/CD (IN PROGRESS)

## Environment Scrubbing
Gopherbot should make every attempt to prevent credential stealing by any potential rogue code. One ugly 'leak' of sensitive data comes from `/proc/\<pid\>/environ` - once a process starts, the contents of `environ` remain unchanged, even after e.g. `os.Unsetenv(...)`. One way to scrub the environment is to use `syscall.Exec`:
```go
package main

import (
	"bufio"
	"fmt"
	"os"
	"syscall"
)

func main() {
	reader := bufio.NewReader(os.Stdin)
	if len(os.Environ()) > 0 {
		fmt.Println("Scrubbing the environment, check my PID/environ and press <enter> to start")
		fmt.Print("-> ")
		_, _ = reader.ReadString('\n')
		self, _ := os.Executable()
		syscall.Exec(self, []string{}, []string{})
	}
	os.Setenv("SCRUBBED", "true")
	fmt.Println("Done. Check PID/environ and press <enter> to finish")
	fmt.Print("-> ")
	_, _ = reader.ReadString('\n')
}
```
When the program is run and prints `Done. ...`, `/proc/\<pid\>/environ` is empty. This algorithm is incomplete, though - now NONE of the environment is available to the new process. The 'fix' for this (needs to be written/tested) is for the initial process to spawn a goroutine that re-runs `gopherbot` with the single argument `export`, and keeps an open FD reading from stdout. The main process can then call `syscall.Exec(...)` and read the env from the FD in to potentially a memguard LockedBuffer, where it can be safely dealt with.

## Robot Configuration
The stuff here needs to be checked / verified. Some may already be done.
To simplify locking:
* Some of the content of gopherbot.yaml should only be processed when the robot first starts, e.g. the brain, connector, listening port, logger
* Start-up items should be stored globally and readable without a lock
* These start-up items should be processed during initbot
* Items that can change on reload should be stored in a config struct similar to plugins and jobs; when the botContext is registered, it should get a copy of this struct that doesn't change for the life of the context, just like the task list
* Items that are processed to binary representations (e.g. string to loglevel) should be stored in non-public struct members; e.g. LogLevel(string) and logLevel(int)

## TODOs:
- DONE/TEST/DOCUMENT: Make sure plugins/tasks/jobs can be disabled in gopherbot.yaml
- TODO: Make connectors pass through e.g. User="<U12345>" when lookup fails
- TODO: Add admin "monitor \<channel\>" / "stop monitoring" to DM admin with all messages to a channel similar to debug, for use in plugin devel & troubleshooting
- TODO: Update ansible role to stop creating installed gopherbot.yaml, favoring version
  from install archive
- TODO: Add Reload() method to connectors, many of which may require additional data
- TODO: Low Priority - Support for Go jobs / tasks (can wait 'til first need)
- DONE/DOCUMENT: try converting lists of tasks/jobs/plugins to map[name]struct
- To obtain the `BXXXXXXXX` ID for a webhook, visit the settings page, e.g.: https://myteam.slack.com/services/BC7DLEPA8 <- that's it!
- TODO: Low Priority - Audit reload cases where something not configured needs to be un-configured, similar to scheduled jobs n->0
- TODO: Add WorkSpace to output from `info`
- TODO: paging for show log similar to history; same 2k page size, but showing in reverse (e.g. 'p' for previous page, 'q' to quit)
- TODO: Low Priority - Brain optimization; keep a global seenCache map[string]bool:
	- if the entry exists, it's been seen and the bool indicates whether the memory exists
	- if the entry doesn't exist, check the memory and store whether it exists
	- when storing a memory, always store that the memory exists
	- best of all, the brain loop should mean no mutex!
- TODO/VERIFY: (This may be complete) Audit use of 'debug' - most calls should be debugT? (before bot.currentTask is set)
- TODO: (LP) Move slack send loop into anon func in Run
- TODO: Documentation for events, emit, development and testing - Developing.md
- DONE/DOCUMENT: Stop hard-coding 'it' for contexts; 'server:it:the server' instead of
  'server', (context followed by generics that match); e.g. "reboot server",
  "reboot the server", "reboot it" would all check the "server" short-term
  memory to get e.g. "webtest1.example.com"
- TODO: This needs more thought - the 'catchall' plugin should be in gopherbot.yaml (There Can Be Only One)<commit_msg>Add TODO<commit_after># Gopherbot Development Notes - Current

`DevNotes.md` - TODO items and design notes for future developmen

## 2.0 Goals
* Job scheduling (DONE)
* Pipelines (DONE)
* Remote plugin execution over ssh (ON HOLD)
* GopherCI for CI/CD (IN PROGRESS)

## Environment Scrubbing
Gopherbot should make every attempt to prevent credential stealing by any potential rogue code. One ugly 'leak' of sensitive data comes from `/proc/\<pid\>/environ` - once a process starts, the contents of `environ` remain unchanged, even after e.g. `os.Unsetenv(...)`. One way to scrub the environment is to use `syscall.Exec`:
```go
package main

import (
	"bufio"
	"fmt"
	"os"
	"syscall"
)

func main() {
	reader := bufio.NewReader(os.Stdin)
	if len(os.Environ()) > 0 {
		fmt.Println("Scrubbing the environment, check my PID/environ and press <enter> to start")
		fmt.Print("-> ")
		_, _ = reader.ReadString('\n')
		self, _ := os.Executable()
		syscall.Exec(self, []string{}, []string{})
	}
	os.Setenv("SCRUBBED", "true")
	fmt.Println("Done. Check PID/environ and press <enter> to finish")
	fmt.Print("-> ")
	_, _ = reader.ReadString('\n')
}
```
When the program is run and prints `Done. ...`, `/proc/\<pid\>/environ` is empty. This algorithm is incomplete, though - now NONE of the environment is available to the new process. The 'fix' for this (needs to be written/tested) is for the initial process to spawn a goroutine that re-runs `gopherbot` with the single argument `export`, and keeps an open FD reading from stdout. The main process can then call `syscall.Exec(...)` and read the env from the FD in to potentially a memguard LockedBuffer, where it can be safely dealt with.

## Robot Configuration
The stuff here needs to be checked / verified. Some may already be done.
To simplify locking:
* Some of the content of gopherbot.yaml should only be processed when the robot first starts, e.g. the brain, connector, listening port, logger
* Start-up items should be stored globally and readable without a lock
* These start-up items should be processed during initbot
* Items that can change on reload should be stored in a config struct similar to plugins and jobs; when the botContext is registered, it should get a copy of this struct that doesn't change for the life of the context, just like the task list
* Items that are processed to binary representations (e.g. string to loglevel) should be stored in non-public struct members; e.g. LogLevel(string) and logLevel(int)

## TODOs:
- DONE/TEST/DOCUMENT: Make sure plugins/tasks/jobs can be disabled in gopherbot.yaml
- TODO: Close job histories after main pipeline; add new task for emailing the job history that can be used in a FinalTask or FailTask
- TODO: Make connectors pass through e.g. User="<U12345>" when lookup fails
- TODO: Add admin "monitor \<channel\>" / "stop monitoring" to DM admin with all messages to a channel similar to debug, for use in plugin devel & troubleshooting
- TODO: Update ansible role to stop creating installed gopherbot.yaml, favoring version
  from install archive
- TODO: Add Reload() method to connectors, many of which may require additional data
- TODO: Low Priority - Support for Go jobs / tasks (can wait 'til first need)
- DONE/DOCUMENT: try converting lists of tasks/jobs/plugins to map[name]struct
- To obtain the `BXXXXXXXX` ID for a webhook, visit the settings page, e.g.: https://myteam.slack.com/services/BC7DLEPA8 <- that's it!
- TODO: Low Priority - Audit reload cases where something not configured needs to be un-configured, similar to scheduled jobs n->0
- TODO: Add WorkSpace to output from `info`
- TODO: paging for show log similar to history; same 2k page size, but showing in reverse (e.g. 'p' for previous page, 'q' to quit)
- TODO: Low Priority - Brain optimization; keep a global seenCache map[string]bool:
	- if the entry exists, it's been seen and the bool indicates whether the memory exists
	- if the entry doesn't exist, check the memory and store whether it exists
	- when storing a memory, always store that the memory exists
	- best of all, the brain loop should mean no mutex!
- TODO/VERIFY: (This may be complete) Audit use of 'debug' - most calls should be debugT? (before bot.currentTask is set)
- TODO: (LP) Move slack send loop into anon func in Run
- TODO: Documentation for events, emit, development and testing - Developing.md
- DONE/DOCUMENT: Stop hard-coding 'it' for contexts; 'server:it:the server' instead of
  'server', (context followed by generics that match); e.g. "reboot server",
  "reboot the server", "reboot it" would all check the "server" short-term
  memory to get e.g. "webtest1.example.com"
- TODO: This needs more thought - the 'catchall' plugin should be in gopherbot.yaml (There Can Be Only One)
{'fraction_non_alphanumeric': 0.09157782515991471, 'fraction_numerical': 0.002771855010660981, 'mean_word_length': 4.154395604395605, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 23, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3166807', 'n_tokens_mistral': 2772, 'n_tokens_neox': 2646, 'n_words': 1320}