<commit_before>Open Questions on Metadata / Schemas / Representations / Validation
===================================================================

## User Defined Metadata Schemas

User defined schema of what's required: items, collections

There is some schema the user wants expressed and enforced.

Should the user's expression and view of the schema be the same as FCMS's internal representation? Ideally, for simplicity, it should, otherwise there is mapping required. But is the mapping more or less work/complexity then validating with the schema as expressed by the user? And is there only one expression/view for the user? Or will FCMS support multiple different ways of representing schemas to/from the user?

User will want to use "off-the-shelf" metadata schemas by just selecting them without having to express them. Casual users will want an "off-the-shelf" schema selected for them.

Relevant standards:
	Dublin Core


### Metadata Schema Representation

Technical representation will be different than UI representation which should be incredibly simple (without being simplistic). But more technical users will care about the technical representation too.

How to externally represent user defined item or collection schema?
	Role of standards?
	Just one or multiple representations?

How to internally represent user defined item or collection schema?
	Same as external? Harder when external is multiple.

Relevant representations:
	Prismatic schema "native"
	XML Schema - [Dublin Core example](http://dublincore.org/schemas/xmls/)
	RDF Schema - [Dublin Core example](http://dublincore.org/schemas/rdfs/)
	JSON?

### Metadata Schema Validation

Create / Update: collection, item
Clojure API / REST API
	Does it have the required items for FCMS?
	Does it not have prohibited items for FCMS?
	Does it have the required items for user schema?

Prismatic schema? https://github.com/prismatic/schema
Validateur? http://clojurevalidations.info/
Manners? https://github.com/RyanMcG/manners

## Taxonomy Schema

The taxonomy schema defines what is and isn't a valid taxonomy. It's not strictly internal as it'll be exposed to API users to define what is and isn't a valid taxonomy. May benefit in terms of cognitive load from using the same technology as item/collection schema.

### Taxonomy Schema Representation

How to represent instance of FCMS taxonomy?<commit_msg>More notes on metadata.<commit_after>Open Questions on Metadata / Schemas / Representations / Validation
===================================================================

## User Defined Metadata Schemas

User defined schema of what's required: items, collections

There is some schema the user wants expressed and enforced.

Should the user's expression and view of the schema be the same as FCMS's internal representation? Ideally, for simplicity, it should, otherwise there is mapping required. But is the mapping more or less work/complexity then validating with the schema as expressed by the user? And is there only one expression/view for the user? Or will FCMS support multiple different ways of representing schemas to/from the user?

User will want to use "off-the-shelf" metadata schemas by just selecting them without having to express them. Casual users will want an "off-the-shelf" schema selected for them.

Relevant standards:
	[Dublin Core](http://dublincore.org/) - "core metadata" for simple and generic resource descriptions
	[Schema.org](http://schema.org/) - an initiative from the big search engines (Google, Bing, Yahoo, Yandex) to index web data

http://www.k4all.ca/book/export/html/189

### Metadata Schema Representation

Technical representation will be different than UI representation which should be incredibly simple (without being simplistic). But more technical users will care about the technical representation too.

There is the representation TOO the tool. Where the user expresses this is the schema I want enforced.

There is the representation FROM the tool. Where the user sees the data values of the schema elements for a particular item they are "looking" at. This could be in an HTML editing page (a web form/UI), an HTML presentation page (RDFa, Microdata, Microformats), a metadat presentation page (RDF) or in an API (JSON, RDF, XML...). [Background](http://stackoverflow.com/questions/14307792/what-is-the-relationship-between-rdf-rdfa-microformats-and-microdata).

How to externally represent user defined item or collection schema?
	Role of standards?
	Just one or multiple representations?

How to internally represent user defined item or collection schema?
	Same as external? Harder when external is multiple.

Relevant representations:
	Prismatic schema "native"
	XML Schema - [Dublin Core example](http://dublincore.org/schemas/xmls/)
	RDF Schema - [Dublin Core example](http://dublincore.org/schemas/rdfs/)
	OWL
	JSON?

### Metadata Schema Validation

Create / Update: collection, item
Clojure API / REST API
	Does it have the required items for FCMS?
	Does it not have prohibited items for FCMS?
	Does it have the required items for user schema?

Prismatic schema? https://github.com/prismatic/schema
Validateur? http://clojurevalidations.info/
Manners? https://github.com/RyanMcG/manners

## Taxonomy Schema

The taxonomy schema defines what is and isn't a valid taxonomy. It's not strictly internal as it'll be exposed to API users to define what is and isn't a valid taxonomy. May benefit in terms of cognitive load from using the same technology as item/collection schema.

### Taxonomy Schema Representation

How to represent instance of FCMS taxonomy?
{'fraction_non_alphanumeric': 0.08206958073148975, 'fraction_numerical': 0.0021409455842997323, 'mean_word_length': 4.839583333333334, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '805773', 'n_tokens_mistral': 1444, 'n_tokens_neox': 1364, 'n_words': 721}