<jupyter_start><jupyter_text>1.x to 3.x Rule Migration GuideThis guide describes changes needed for rules to run under Insights Core 3.x. It covers the following topics:- filtering- decorator interfaces- function signatures- cluster rules- testing- new style specs  FilteringFilters are now applied to registry points or datasources instead of certain `Parser` classes.```pythonfrom insights.specs import Specsfrom insights.core.filters import add_filter do thisadd_filter(Specs.messages, "KEEP_ME")add_filter(Specs.messages, ["KEEUP_US", "KEEP_US_TOO"]) instead of thisMessages.filters.append("KEEP_ME")Messages.filters.extend(["KEEP_US", "KEEP_US_TOO"])```  Decorator InterfacesThe `requires` keyword is gone, and required dependencies are no longer lists.```python@rule(requires=[InstalledRpms, PsAuxcww])```is now```python requires InstalledRpms and PsAuxcww@rule(InstalledRpms, PsAuxcww)```If a rule requires at least one of a set of dependencies, they are specified in a list like before.```python requires InstalledRpms and at least one of ChkConfig or UnitFiles@rule(InstalledRpms, [ChkConfig, UnitFiles])```And `optional` dependencies haven't changed.```python requires InstalledRpms and PsAuxcww. Will use NetstatS if it's available@rule(InstalledRpms, PsAuxcww, optional=[NetstatS])```  Component SignatureThe `local` and `shared` parameters are gone. Instead, component signatures should define parameters matching the dependencies in their decorators.```python Requires InstalledRpms and PsAuxcww.@rule(InstalledRpms, PsAuxcww)def report_thing(rpms, ps):    pass Requires InstalledRpms and at least one of ChkConfig or UnitFiles. Both ChkConfig and UnitFiles may be populated, but only one of them is required. If one of them isn't available, None is passed as its value.@rule(InstalledRpms, [ChkConfig, UnitFiles])def report_something(rpms, cfg, uf):    pass Requires InstalledRpms, at least one of ChkConfig or UnitFiles, and will use NetstatS if it's available. Notice how the order of report_something_else's parameter list matches the order of the dependencies even when the dependency specification is complicated.@rule(InstalledRpms, [ChkConfig, UnitFiles], optional=[NetstatS])def report_something_else(rpms, cfg, uf, netstat):    pass```  ConditionsCondition behavior has been standardized to match the behavior of every other component. This means that to remove a condition from the dependency chain an exception must be raised.The purpose of a condition is to determine whether something is `True` or `False` and surface that information to other systems. The values they generate, other than raised exceptions, will always be passed on to components that depend on them.```python@condition(Messages)def has_line(msgs):    return "interesting_line" in msgs:```Note that `True` or `False` is returned explicitly instead of implicitly returning `None`.**It's important to remember that a condition shouldn't return `None` when it means `False`**.    That's because `None` signals "Could not determine whether True or False" to other systems. For example:```python@condition(SomeConfig)def check_bar_greater_than_10(config):    if "bar" in config:        return int(config["bar"]) > 10```What's the appropriate boolean value for `check_bar_greater_than_10` if "bar" doesn't exist in `config`? It could be treated as `False`, but the value isn't actually there, and if we don't know the default value, the condition can't really say either way.At this point, you could do one of two things depending on whether you want the rule to still execute.You could raise a `dr.SkipComponent()` instead of returning `None`. External systems will still treat the condition's value as `None`. If the condition is required by the rule, the rule won't execute. If the condition is optional for the rule, the rule will still execute but will be passed `None` for the condition's value.If the condition returns `None`, external systems will treat its value as `None`, and the dependent rule will execute whether the dependency is required or optional.For example, the following rule will always execute if the condition's dependencies are met and it doesn't raise an exception regardless of what the condition returns:```python@rule(check_bar_greater_than_10)def my_rule(has_line):    if has_line:   could be True, False, None, etc.        return make_response("ERROR")```If the condition was written like this instead and the rule was the same, the rule would not execute:```python@condition(SomeConfig)def check_bar_greater_than_10(config):    if "bar" in config:        return int(config["bar"]) > 10      we raise an exception. Since the rule requires us, it doesn't execute.      External system treat our value as None.    raise dr.SkipComponent()```However, if the rule was written like this, it would always execute regardless of what the condition returned or exceptions it raised. If the condition returns something, it gets passed to the rule. If it raises an exception, `None` gets passed to the rule.```python@rule(optional=[check_bar_greater_than_10])def my_rule(has_line):    if has_line:        return make_response("ERROR")```<jupyter_code># Boilerplate used in later cells
# Not necessary for new rules.

import sys
sys.path.append("..")

from pprint import pprint

from insights import run
from insights.core.dr import load_components
from insights.core.plugins import make_response, rule

load_components("insights.specs.default")
load_components("insights.specs.insights_archive")
load_components("insights.specs.sos_archive")
load_components("insights.parsers")
load_components("insights.combiners")<jupyter_output><empty_output><jupyter_text>@rule Example<jupyter_code>from insights.parsers.installed_rpms import InstalledRpms
from insights.parsers.ps import PsAuxcww

@rule(InstalledRpms, PsAuxcww)
def report(rpms, ps):
    rpm_name = "bash"
    if rpm_name in rpms and "bash" in ps:
        rpm = rpms.get_max(rpm_name)
        return make_response("BASH_RUNNING",
                             version=rpm.version,
                             release=rpm.release,
                             arch=rpm.arch
                            )
broker = run(report)
pprint(broker[report])<jupyter_output>{'arch': 'x86_64',
 'error_key': 'BASH_RUNNING',
 'release': '13.fc27',
 'type': 'rule',
 'version': '4.4.12'}<jupyter_text>Cluster RulesCluster rules are not supported at this time. You can mark a rule as a cluster rule by adding `cluster=True` to its decorator, but these rules don't actually run since cluster archives are not supported.  TestingUnit tests need to reflect the new function signatures of condition, rules, etc. Rules no longer take `local` and `shared` parameters but are instead directly passed their dependencies as arguments. Integration Tests`InputData.add` now takes `Specs.` as its first parameter instead of a string. You'll need to add `from insights.specs import Specs` to the import list at the top of the test file.The `yield input_data, [expected]` lines should instead be `yield input_data, expected`. Also, yield `None` instead of `[]`. In summary, there's no need to wrap the expected result in a list.`@archive_provider` accepts the rule itself instead of the rule module:```pythonfrom insights.plugins import vulnerable_kernel do this@archive_provider(vulnerable_kernel.report)def integration_tests():...instead of this@archive_provider(vulnerable_kernel)def integration_tests():...```  New Style SpecsSpecs in 3.x are called "datasources", and they're functions like rules and other components. However, they are special because they get passed an object called a `broker` instead of directly getting their dependencies, and they're meant to execute on the machine you want to analyze. The `broker` is like the `shared` object in 1.x.  A much more detailed guide on the new spec system can be found in the [Datasource Registry](https://github.com/RedHatInsights/insights-core/blob/master/notebooks/Datasource%20Registry.ipynb) notebook.<jupyter_code>from insights.core.plugins import datasource
from insights.core.spec_factory import SpecSet, TextFileProvider

class TheSpecs(SpecSet):

    @datasource()
    def release(broker):
        return TextFileProvider("etc/redhat-release")
broker = run(TheSpecs.release)
print broker[TheSpecs.release].content<jupyter_output>['Fedora release 27 (Twenty Seven)']<jupyter_text>This allows data sources to generate content using the full power of python. Almost anything can go in the function body of a data source. Directly defining data sources is powerful, but it's tedious when you just want to collect files or execute simple commands. The `spec_factory` module streamlines those use cases by creating `@datasource` decorated functions for you.<jupyter_code>from insights.specs import Specs
from insights.core.spec_factory import simple_file, simple_command

class MySpecs(Specs):
    hosts = simple_file("/etc/hosts")
    uptime = simple_command("/bin/uptime")


print MySpecs.hosts
print MySpecs.uptime
broker = run(MySpecs.hosts)
pprint(broker[MySpecs.hosts].content)

broker = run(MySpecs.uptime)
pprint(broker[MySpecs.uptime].content)<jupyter_output>['127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4',
 '::1         localhost localhost.localdomain localhost6 localhost6.localdomain6']
[' 10:29:40 up 18 days, 20:05,  1 user,  load average: 1.01, 0.94, 1.00']
{'fraction_non_alphanumeric': 0.08175632574952159, 'fraction_numerical': 0.00744205826068467, 'mean_word_length': 5.300736771600803, 'pattern_counts': {'":': 0, '<': 14, '<?xml version=': 0, '>': 16, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '70326', 'license': 'Apache-2.0', 'n_tokens_mistral': 2683, 'n_tokens_neox': 2522, 'n_words': 1053, 'path': 'docs/notebooks/Rule Migration Guide.ipynb', 'repo_name': 'dougsland/insights-core', 'size': '9406'}