<jupyter_start><jupyter_text>`optimade-python-tools` https://github.com/Materials-Consortia/optimade-python-tools Active maintainers (alphabetical):- <NAME>, <NAME>, <NAME>, <NAME> Many more contributors (see [GitHub contributors](https://github.com/Materials-Consortia/optimade-python-tools/graphs/contributors)):  - <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, <NAME>   What is it?- An extensible reference server implementation that uses MongoDB/`mongomock` as a backend (both regular and index meta-db servers)- Type-hinted Python models that describe entries and responses of an OPTIMADE-compliant API (using [`fastapi`](https://fastapi.tiangolo.com/) and [`pydantic`](https://pydantic-docs.helpmanual.io))- [`lark`](https://lark-parser.readthedocs.io/en/latest/)-format description of the filter language grammar with associated parsers.- OPTIMADE implementation validator, with associated [GitHub Action](https://github.com/Materials-Consortia/optimade-validator-action).  DisclaimerAlthough we are nearly 1.0 spec compliant, the actual optimade-python-tools API will probably change and become more usable in the coming weeks/months.  Who/what is it for?- Database providers:    - New implementations    - Existing implementations who want a simple way of running the index meta-db- Specification development:     - Useful for quickly prototyping changes to the spec- Validation of existing implementations  Who is using it?- Materials Cloud: https://www.materialscloud.org/optimade- Materials Project: https://optimade.materialsproject.org/- odbx: https://optimade.odbx.science- NOMAD? OQMD?  Explore the GitHub:- Source: https://github.com/Materials-Consortia/optimade-python-tools- Docs (in progress): https://www.optimade.org/optimade-python-tools/  Anatomy of a query<jupyter_code>display(SVG(filename="./query.svg"))<jupyter_output><empty_output><jupyter_text>Creating a new implementation - if you think there is a simple map* between your database and OPTIMADE properties:    - might be able to get away with no extra code, just configuration (see later)- else:    - custom mappers: `optimade.server.mappers.StructureMapper` - if you're using MongoDB:     - hopefully you can use our transformers- else:    - inherit `optimade.server.entry_collections.EntryCollection`    - inherit `optimade.server.filter_transformers.BaseTransformer` \* simple map: field name changes, cheap operations on the underlying fields. Does not necessarily guarantee queryability!e.g. `"elements": ["Na", "Cl"]` vs `my_elements: "Na,Cl"`: can transform response with default `StructureMapper`, but custom filter transformer needed.  Advantages- Can be used to very quickly prototype an implementation, especially if already using MongoDB- Quirks of filtering and responses are handled automatically- Changes to the specification are reflected quickly by the community- Modularity; use the bits you need.  Drawbacks- Without the underlying fields in the database, mapping responses is often easy, but mapping *queries* can be very difficult.    - specification currently mandates all standardized properties must be queryable- New backends with a great deal of customisation require more work, but hopefully still less than starting from scratch- Scalability: as Shyam mentioned. How many implementations can handle the load of MP/NOMAD/COD?  Pydantic models- Type-hinted Python classes for with validators that define an OpenAPI schema:    - entries    - responses    - requests  Lark grammar- Rewritten by Adam after the last workshop- Handles everything in the specification, with a relatively easy interface to define a new transformer  Intra-resource property queries- Length aliases:    - Perform OPTIMADE `LENGTH` query indirectly using a second field, e.g. `LENGTH species_at_sites > 2 -> nsites > 2`  Plans for CECAM workshop 2020- Improve documentation and general usability- Get feedback and get new implementations and contributors involved [GitHub "good first issue"](https://github.com/Materials-Consortia/optimade-python-tools/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22)- Tackle known issues blocking release for 1.0 specification [GitHub CECAM project board](https://github.com/Materials-Consortia/optimade-python-tools/projects/2)- Discuss future of the project (Room 4) Contributing- Make sure you're in the Materials Consortia GitHub organization- Raise issues to discuss what you are planning to do (or comment on an existing issue indicating that you want to work on it)- Push a feature branch to the main repository and make a pull request!   Switch to Casper's Clienthttps://github.com/aiidalab/aiidalab-optimade/  `optimade-python-tools` Demos   Configuration-only implementation from reference Example config: https://github.com/Materials-Consortia/optimade-python-tools/blob/master/example_config.json ```json{    "debug": false,    "use_real_mongo": true,    "mongo_database": "optimade",    "mongo_uri": "mongodb+srv://MongoUser:MongoPassword@FakeMongoServer.com/optimade_database",    "links_collection": "optimade_links",    "references_collection": "optimade_references",    "structures_collection": "optimade_structures",    "default_db": "example_server",    "base_url": "https://optimade.example.org",    "implementation": {        "name": "Example implementation",        "source_url": "https://github.com/Materials-Consortia/optimade-python-tools",        "maintainer": {"email": "<EMAIL>"}    },    "provider": {        "name": "Example provider",        "description": "Provider used for examples, not to be assigned to a real database",        "prefix": "exmpl",        "homepage": "https://example.com",        "index_base_url": "https://example.com/index"    },    "provider_fields": {        "structures": [            "band_gap",            "chemsys"        ]    },    "aliases": {        "structures": {            "id": "task_id",            "chemical_formula_descriptive": "pretty_formula",            "chemical_formula_reduced": "pretty_formula",            "chemical_formula_anonymous": "formula_anonymous"        }    },    "length_aliases": {        "structures": {            "chemsys": "nelements"        }    }}``` Example structure:<jupyter_code>!cat structures.json<jupyter_output>[
  {
    "dimension_types": [1, 1, 1],
    "elements": ["Ac"],
    "elements_ratios": [1.0],
    "formula_anonymous": "A",
    "last_modified": {"$date": "2019-06-08T05:13:37.331Z"},
    "lattice_vectors": [
        [1.2503264826932692, 0.0, 0.0],
        [0.0, 9.8885097163217651, 0.0],
        [0.0, 0.0, 10.972637673241818]
    ],
    "nelements": 1,
    "pretty_formula": "Ac",
    "species": [{"chemical_symbols": ["Ac"], "concentration": [1.0], "name": "Ac"}],
    "species_at_sites": ["Ac"],
    "relationships": {"references": {"data": [{"type": "references", "id": "dijkstra1968"}]}},
    "structure_features": [],
    "num_atoms": 1,
    "positions_cart": [[0.0, 1.0, 2.0]],
    "chemsys": "Ac",
    "task_id": "mpf_1",
    "band_gap": 1.23,
    "_id": {
      "$oid": "5cfb441f053b174410700d02"
    }
  }
]<jupyter_text>Extending the pydantic models and app code<jupyter_code>from pydantic import Field, validator
from typing import Dict, List
from optimade.models import StructureResourceAttributes, StructureResource
from optimade.server.mappers import StructureMapper

class MyStructureResourceAttributes(StructureResourceAttributes):
    
    entropy: float = Field(
        ..., description="Entropic contribution to total energy"
    )
    pseudopotentials: Dict[str, str] = Field(
        ..., description="The pseudopotentials used for each species in the calculation"
    )
    
    @validator("entropy")
    def check_entropy(cls, v):
        assert v >= 0.0, "warning: 2nd law of thermodynamics broken"
        
class MyStructureResource(StructureResource):
    
    attributes: MyStructureResourceAttributes
        
class MyStructureMapper(StructureMapper):
   
    PROVIDER_FIELDS = {"entropy", "pseudopotentials"}
    
    def map_back(*args, **kwargs): 
        doc = super().map_back(*args, **kwargs)
        doc['chemical_formula_anonymous'] = form2anon(doc['chemical_formula_reduced'])
import datetime
na_cl = MyStructureResourceAttributes(
    lattice_vectors=[[4, 4, 0], [4, 0, 4], [0, 4, 4]],
    last_modified=datetime.datetime.now(),
    elements=["Cl", "Na"],
    nelements=2,
    elements_ratios=[0.5, 0.5],
    chemical_formula_descriptive="ClNa",
    chemical_formula_anonymous="AB",
    chemical_formula_reduced="ClNa",
    dimension_types=(1, 1, 1),
    nsites=2,
    cartesian_site_positions=((0, 0, 0), (2, 2, 2)),
    species_at_sites=["Na", "Cl"],
    species=[{"name": "Na", "chemical_symbols": ["Na"], "concentration": [1]},
             {"name": "Cl", "chemical_symbols": ["Cl"], "concentration": [1]}
            ],
    structure_features=[],
    entropy=10,
    pseudopotentials={"Na": "Na.usp", "Cl": "Cl.usp"}
)
na_cl.schema()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12441627751834557, 'fraction_numerical': 0.01912386035134534, 'mean_word_length': 4.190421234852856, 'pattern_counts': {'":': 67, '<': 25, '<?xml version=': 0, '>': 29, 'https://': 19, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '446841', 'license': 'MIT', 'n_tokens_mistral': 2803, 'n_tokens_neox': 2561, 'n_words': 766, 'path': 'optimade_python_tools_demo.ipynb', 'repo_name': 'ml-evs/optimade-python-tools-demo', 'size': '9100'}