<commit_before># cmr-metadata-db-app

## Temporary Notes on id generation

TODO move this stuff to a more permanent home in the README

  * get /concept-id should return a 404 if it doesn't exist in the concepts table
  * There should only be a concepts table
  * Concepts table should have a uniqueness constraint on
    * provider ids and revision id
      * provider-id
      * native-id
      * concept-type
      * revision-id
    * concept id and revision id

If there's a conflict on provider ids and revision id uniqueness then we need to retry the transaction.

General Workflow

Update Flow

  - Retrieve latest revision from DB using provider-id, concept-type, and native id.
  - Compare revision from client if given to DB revision. If the revision from the client is not the next one we send a conflict error back to the client.
  - Create a new concept record
    - increment revision from DB
    - Reuse concept-id
    - Set all other fields
  - Insert into table
  - If we get a conflict from a uniqueness constraint restart from beginning of this flow

Insert Flow

  - Retrieve latest revision from DB (and none are found)
  - Check if revision id sent by client is 0 if present. If the revision from the client is not 0 we send a conflict error back to the client.
  - Create a new concept record
    - Revision is 0
    - Generate a new concept-id using a sequence from Oracle or use value from client if provided.
      - This supports catalog rest specifying the concept-id.
    - Set all other fields
  - Insert into table
  - If we get a conflict from a uniqueness constraint restart from beginning of this flow


Impacts to Ingest

  * Ingest flow (after phase 1)
    * retrieve existing metadata
    * Use it for validation with new metadata
    * Send new concept with specified revision id to metadata db
      * If it fails then it would redo the whole flow.
  * Ingest Flow (during phase 1)
    * Send new concept to metadata db
      * Does not need to specify a revision id.
      * Does not need to retrieve a concept id.
        * It should check it's headers for a concept id from catalog rest. If present it should send it to the metadata db.


## Web API

### Sample Concept JSON
	{
		"concept-type": "collection"
   	"native-id": "provider collection id"
   	"concept-id": "C1-PROV1"
   	"provider-id": "PROV1"
   	"metadata": "xml here"
   	"format": "echo10"
   	"revision-id": 1 (optional field)
	}

### Sample Tombstone (deleted concept) JSON
	{
		"concept-type": "collection"
		"native-id": "provider collection id"
   	"concept-id": "C1-PROV1"
   	"provider-id": "PROV1"
   	"deleted": true
   	"revision-id": 10
   }

General Workflow

Update Flow

  - Retrieve latest revision from DB using provider-id, concept-type, and native id.
  - Compare revision from client if given to DB revision. If the revision from the client is not the next one we send a conflict error back to the client.
  - Create a new concept record
    - increment revision from DB
    - Reuse concept-id
    - Set all other fields
  - Insert into table
  - If we get a conflict from a uniqueness constraint restart from beginning of this flow

Insert Flow

  - Retrieve latest revision from DB (and none are found)
  - Check if revision id sent by client is 0 if present. If the revision from the client is not 0 we send a conflict error back to the client.
  - Create a new concept record
    - Revision is 0
    - Generate a new concept-id using a sequence from Oracle or use value from client if provided.
      - This supports catalog rest specifying the concept-id.
    - Set all other fields
  - Insert into table
  - If we get a conflict from a uniqueness constraint restart from beginning of this flow


### GET /concept-id
params: [concept-type provider-id native-id]
returns: new or existing concept-id

__Example Curl:__
curl -v http://localhost:3001/concept-id/collection/PROV1/native-id

### POST /concepts
params: [concept] - revision-id optionally in concept
returns: revision-id.  revision-id begins at 0.
throws error if revision-id does not match what it will be when saved

__Example Curl:__
curl -v -XPOST -H "Content-Type: application/json" -d '{"concept-type": "collection", "native-id": "native-id", "concept-id": "C1-PROV1", "provider-id": "PROV1", "metadata": "xml here", "format": "echo10"}' http://localhost:3001/concepts/

### GET /concepts/#concept-id
params: none
returns: latest revision of a concept with the given concept-id

__Example Curl:__
curl -v http://localhost:3001/concepts/C1-PROV1

### GET /concepts/#concept-id/#revision-id
params: none
returns: concept with the given concept-id and revision-id

__Example Curl:__
curl -v http://localhost:3001/concepts/C1-PROV1/2

### POST /concepts/search
params: [{"concept-revisions": [concept-id/revision-id tuple] ...]}]
returns: list of concepts matching the touples provided in the body of the POST

__Example Curl:__
curl -v -XPOST -H "Content-Type: application/json" -d '{"concept-revisions": [["C1-PROV1", 1], ["C2-PROV1", 1]]}' http://localhost:3001/concepts/search

### DELETE /concepts/#concept-id
params: none
returns: the revision id of the tombstone generated for the concept

__Example Curl:__
curl -v -XDELETE localhost:3001/concepts/C1-PROV1

### DELETE /concepts/force-delete/:concept-id/:revision-id
params: none
returns: nothing (status 204)

__Example Curl:__
curl -v -XDELETE /concepts/force-delet/C1-PROV1/1

### POST /reset
params: none<commit_msg>CMR-132: Added example for force-delete to README.md<commit_after># cmr-metadata-db-app

## Temporary Notes on id generation

TODO move this stuff to a more permanent home in the README

  * get /concept-id should return a 404 if it doesn't exist in the concepts table
  * There should only be a concepts table
  * Concepts table should have a uniqueness constraint on
    * provider ids and revision id
      * provider-id
      * native-id
      * concept-type
      * revision-id
    * concept id and revision id

If there's a conflict on provider ids and revision id uniqueness then we need to retry the transaction.

General Workflow

Update Flow

  - Retrieve latest revision from DB using provider-id, concept-type, and native id.
  - Compare revision from client if given to DB revision. If the revision from the client is not the next one we send a conflict error back to the client.
  - Create a new concept record
    - increment revision from DB
    - Reuse concept-id
    - Set all other fields
  - Insert into table
  - If we get a conflict from a uniqueness constraint restart from beginning of this flow

Insert Flow

  - Retrieve latest revision from DB (and none are found)
  - Check if revision id sent by client is 0 if present. If the revision from the client is not 0 we send a conflict error back to the client.
  - Create a new concept record
    - Revision is 0
    - Generate a new concept-id using a sequence from Oracle or use value from client if provided.
      - This supports catalog rest specifying the concept-id.
    - Set all other fields
  - Insert into table
  - If we get a conflict from a uniqueness constraint restart from beginning of this flow


Impacts to Ingest

  * Ingest flow (after phase 1)
    * retrieve existing metadata
    * Use it for validation with new metadata
    * Send new concept with specified revision id to metadata db
      * If it fails then it would redo the whole flow.
  * Ingest Flow (during phase 1)
    * Send new concept to metadata db
      * Does not need to specify a revision id.
      * Does not need to retrieve a concept id.
        * It should check it's headers for a concept id from catalog rest. If present it should send it to the metadata db.


## Web API

### Sample Concept JSON
	{
		"concept-type": "collection"
   	"native-id": "provider collection id"
   	"concept-id": "C1-PROV1"
   	"provider-id": "PROV1"
   	"metadata": "xml here"
   	"format": "echo10"
   	"revision-id": 1 (optional field)
	}

### Sample Tombstone (deleted concept) JSON
	{
		"concept-type": "collection"
		"native-id": "provider collection id"
   	"concept-id": "C1-PROV1"
   	"provider-id": "PROV1"
   	"deleted": true
   	"revision-id": 10
   }

General Workflow

Update Flow

  - Retrieve latest revision from DB using provider-id, concept-type, and native id.
  - Compare revision from client if given to DB revision. If the revision from the client is not the next one we send a conflict error back to the client.
  - Create a new concept record
    - increment revision from DB
    - Reuse concept-id
    - Set all other fields
  - Insert into table
  - If we get a conflict from a uniqueness constraint restart from beginning of this flow

Insert Flow

  - Retrieve latest revision from DB (and none are found)
  - Check if revision id sent by client is 0 if present. If the revision from the client is not 0 we send a conflict error back to the client.
  - Create a new concept record
    - Revision is 0
    - Generate a new concept-id using a sequence from Oracle or use value from client if provided.
      - This supports catalog rest specifying the concept-id.
    - Set all other fields
  - Insert into table
  - If we get a conflict from a uniqueness constraint restart from beginning of this flow


### GET /concept-id
params: [concept-type provider-id native-id]
returns: new or existing concept-id

__Example Curl:__
curl -v http://localhost:3001/concept-id/collection/PROV1/native-id

### POST /concepts
params: [concept] - revision-id optionally in concept
returns: revision-id.  revision-id begins at 0.
throws error if revision-id does not match what it will be when saved

__Example Curl:__
curl -v -XPOST -H "Content-Type: application/json" -d '{"concept-type": "collection", "native-id": "native-id", "concept-id": "C1-PROV1", "provider-id": "PROV1", "metadata": "xml here", "format": "echo10"}' http://localhost:3001/concepts/

### GET /concepts/#concept-id
params: none
returns: latest revision of a concept with the given concept-id

__Example Curl:__
curl -v http://localhost:3001/concepts/C1-PROV1

### GET /concepts/#concept-id/#revision-id
params: none
returns: concept with the given concept-id and revision-id

__Example Curl:__
curl -v http://localhost:3001/concepts/C1-PROV1/2

### POST /concepts/search
params: [{"concept-revisions": [concept-id/revision-id tuple] ...]}]
returns: list of concepts matching the touples provided in the body of the POST

__Example Curl:__
curl -v -XPOST -H "Content-Type: application/json" -d '{"concept-revisions": [["C1-PROV1", 1], ["C2-PROV1", 1]]}' http://localhost:3001/concepts/search

### DELETE /concepts/#concept-id/#revision-id
params: none
returns: the revision id of the tombstone generated for the concept

__Example Curl:__
curl -v -XDELETE localhost:3001/concepts/C1-PROV1/1

### DELETE /concepts/#concept-id
params: none
returns: the revision id of the tombstone generated for the concept

__Example Curl:__
curl -v -XDELETE localhost:3001/concepts/C1-PROV1

### DELETE /concepts/force-delete/:concept-id/:revision-id
params: none
returns: nothing (status 204)

__Example Curl:__
curl -v -XDELETE /concepts/force-delet/C1-PROV1/1

### POST /reset
params: none
{'fraction_non_alphanumeric': 0.08316937935968521, 'fraction_numerical': 0.013772133786442497, 'mean_word_length': 3.3462883793237466, 'pattern_counts': {'":': 42, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 4}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6729618', 'n_tokens_mistral': 3422, 'n_tokens_neox': 3207, 'n_words': 1478}