geonature.core.imports.checks.sql

Submodules

• geonature.core.imports.checks.sql.core
• geonature.core.imports.checks.sql.extra
• geonature.core.imports.checks.sql.geo
• geonature.core.imports.checks.sql.nomenclature
• geonature.core.imports.checks.sql.parent
• geonature.core.imports.checks.sql.user
• geonature.core.imports.checks.sql.utils

Classes

ImportCodeError	List of all the possible errors returned during the import process.
BibFields
Entity
TImports

Functions

init_rows_validity(imprt[, dataset_name_field])	Validity columns are three-states:
check_orphan_rows(imprt)
do_nomenclatures_mapping(→ None)	Set nomenclatures using content mapping.
check_nomenclature_exist_proof(→ None)	Check the existence of a nomenclature proof in the transient table.
check_nomenclature_blurring(imprt, entity, ...)	Raise an error if blurring not set.
check_nomenclature_source_status(→ None)	Check the nomenclature source status and raise an error if the status is "Lit" (Literature)
check_nomenclature_technique_collect(→ None)	Check the nomenclature source status and raise an error if the status is "Autre, préciser"
set_geom_point(→ None)	Set the_geom_point as the centroid of the geometry in the transient table of an import.
convert_geom_columns(→ None)	Convert the geometry from the file SRID to the local SRID in the transient table of an import.
check_is_valid_geometry(→ None)	Check if the geometry is valid in the transient table of an import.
check_geometry_outside(→ None)	For an import, check if one or more geometries in the transient table are outside a defined area.
get_duplicates_query(imprt, dest_field[, whereclause])
report_erroneous_rows(imprt, entity, error_type, ...)	Report erroneous rows in a transient table and update validity of rows based on error level.
check_referential(→ None)	Check the referential integrity of a column in the transient table.
check_cd_nom(→ None)	Check the existence of a cd_nom in the Taxref referential.
check_cd_hab(→ None)	Check the existence of a cd_hab in the Habref referential.
generate_altitudes(→ None)	Generate the altitudes based on geomatries, and given altitues in an import.
check_duplicate_uuid(imprt, entity, uuid_field)	Check if there is already a record with the same uuid in the transient table. Include an error in the report for each entry with a uuid dupplicated.
check_existing_uuid(imprt, entity, uuid_field[, ...])	Check if there is already a record with the same uuid in the destination table.
generate_missing_uuid_for_id_origin(imprt, uuid_field, ...)	Update records in the transient table where the uuid is None
generate_missing_uuid(imprt, entity, uuid_field[, ...])	Update records in the transient table where the UUID is None
check_duplicate_source_pk(→ None)	Check for duplicate source primary keys in the transient table of an import.
check_dates(→ None)	Check the validity of dates in the transient table of an import.
check_altitudes(→ None)	Check the validity of altitudes in the transient table of an import.
check_depths(→ None)	Check the validity of depths in the transient table of an import.
check_digital_proof_urls(imprt, entity, ...)	Checks for valid URLs in a given column of a transient table.
check_entity_data_consistency(imprt, entity, fields, ...)	Checks for rows with the same uuid, but different contents,
disable_duplicated_rows(imprt, entity, fields, ...)	When several rows have the same value in grouping field (typically UUID) field,
generate_entity_id(→ None)	Generate the id for each new valid entity
set_parent_id_from_line_no(→ None)	Set the parent id of each entity in the transient table using the line_no of the corresponding parent.
set_id_parent_from_destination(→ None)	Complete the id_parent column in the transient table of an import when the parent already exists in the destination table.
set_parent_line_no(→ None)	Set parent_line_no on child entities when:
check_no_parent_entity(→ None)	Station may be referenced:
check_erroneous_parent_entities(→ None)	Check for erroneous (not valid) parent entities in the transient table of an import.

Package Contents

geonature.core.imports.checks.sql.init_rows_validity(imprt: geonature.core.imports.models.TImports, dataset_name_field: str = 'id_dataset')

Validity columns are three-states:

• None: the row does not contains data for the given entity

• False: the row contains data for the given entity, but data are erroneous

• True: the row contains data for the given entity, and data are valid

geonature.core.imports.checks.sql.check_orphan_rows(imprt: geonature.core.imports.models.TImports)

geonature.core.imports.checks.sql.do_nomenclatures_mapping(imprt: geonature.core.imports.models.TImports, entity: geonature.core.imports.models.Entity, fields: Mapping[str, geonature.core.imports.models.BibFields], fill_with_defaults: bool = False) → None

Set nomenclatures using content mapping.

Parameters

imprtTImports

The import to check.

entityEntity

The entity to check.

fieldsMapping[str, BibFields]

Mapping of field names to BibFields objects.

fill_with_defaultsbool, optional

If True, fill empty user fields with default nomenclatures.

Notes

See the following link for explanation on empty fields and default nomenclature handling: https://github.com/PnX-SI/gn_module_import/issues/68#issuecomment-1384267087

geonature.core.imports.checks.sql.check_nomenclature_exist_proof(imprt: geonature.core.imports.models.TImports, entity: geonature.core.imports.models.Entity, nomenclature_field: geonature.core.imports.models.BibFields, digital_proof_field: geonature.core.imports.models.BibFields | None, non_digital_proof_field: geonature.core.imports.models.BibFields | None) → None

Check the existence of a nomenclature proof in the transient table.

Parameters

imprtTImports

The import to check.

entityEntity

The entity to check.

nomenclature_fieldBibFields

The field representing the nomenclature to check.

digital_proof_fieldOptional[BibFields]

The field for digital proof, if any.

non_digital_proof_fieldOptional[BibFields]

The field for non-digital proof, if any.

geonature.core.imports.checks.sql.check_nomenclature_blurring(imprt, entity, blurring_field, id_dataset_field)

Raise an error if blurring not set. Required if the dataset is private.

geonature.core.imports.checks.sql.check_nomenclature_source_status(imprt: geonature.core.imports.models.TImports, entity: geonature.core.imports.models.Entity, source_status_field: geonature.core.imports.models.BibFields, ref_biblio_field: geonature.core.imports.models.BibFields) → None

Check the nomenclature source status and raise an error if the status is « Lit » (Literature) whereas the reference biblio field is empty.

Parameters

imprtTImports

The import to check.

entityEntity

The entity to check.

source_status_fieldBibFields

The field representing the source status.

ref_biblio_fieldBibFields

The field representing the reference bibliography.

Notes

The error codes are:

• CONDITIONAL_MANDATORY_FIELD_ERROR: the field is mandatory and not set.

geonature.core.imports.checks.sql.check_nomenclature_technique_collect(imprt: geonature.core.imports.models.TImports, entity: geonature.core.imports.models.Entity, source_status_field: geonature.core.imports.models.BibFields, technical_precision_field: geonature.core.imports.models.BibFields) → None

Check the nomenclature source status and raise an error if the status is « Autre, préciser » whereas technical precision field is empty.

Parameters

imprtTImports

The import to check.

entityEntity

The entity to check.

source_status_fieldBibFields

The field representing the source status.

technical_precision_fieldBibFields

The field representing the technical precision.

Notes

The error codes are:

• CONDITIONAL_MANDATORY_FIELD_ERROR: the field is mandatory and not set.

geonature.core.imports.checks.sql.set_geom_point(imprt: geonature.core.imports.models.TImports, entity: geonature.core.imports.models.Entity, geom_4326_field: geonature.core.imports.models.BibFields, geom_point_field: geonature.core.imports.models.BibFields) → None

Set the_geom_point as the centroid of the geometry in the transient table of an import.

Parameters

imprtTImports

The import to update.

entityEntity

The entity to update.

geom_4326_fieldBibFields

Field containing the geometry in the transient table.

geom_point_fieldBibFields

Field to store the centroid of the geometry in the transient table.

Returns

None

geonature.core.imports.checks.sql.convert_geom_columns(imprt: geonature.core.imports.models.TImports, entity: geonature.core.imports.models.Entity, geom_4326_field: geonature.core.imports.models.BibFields, geom_local_field: geonature.core.imports.models.BibFields) → None

Convert the geometry from the file SRID to the local SRID in the transient table of an import.

Parameters

imprtTImports

The import to update.

entityEntity

The entity to update.

geom_4326_fieldBibFields

Field representing the geometry in the transient table in SRID 4326.

geom_local_fieldBibFields

Field representing the geometry in the transient table in the local SRID.

geonature.core.imports.checks.sql.check_is_valid_geometry(imprt: geonature.core.imports.models.TImports, entity: geonature.core.imports.models.Entity, wkt_field: geonature.core.imports.models.BibFields, geom_field: geonature.core.imports.models.BibFields) → None

Check if the geometry is valid in the transient table of an import.

Parameters

imprtTImports

The import to check.

entityEntity

The entity to check.

wkt_fieldBibFields

Field containing the source WKT of the geometry.

geom_fieldBibFields

Field containing the geometry from the WKT in wkt_field to be validated.

geonature.core.imports.checks.sql.check_geometry_outside(imprt: geonature.core.imports.models.TImports, entity: geonature.core.imports.models.Entity, geom_local_field: geonature.core.imports.models.BibFields, id_area: int) → None

For an import, check if one or more geometries in the transient table are outside a defined area.

Parameters

imprtTImports

The import to check.

entityEntity

The entity to check.

geom_local_fieldBibFields

Field containing the geometry in the local SRID of the area.

id_areaint

The id of the area to check if the geometry is inside.

class geonature.core.imports.checks.sql.ImportCodeError

List of all the possible errors returned during the import process.

Attributes

DATASET_NOT_FOUNDstr

The referenced dataset was not found

DATASET_NOT_AUTHORIZEDstr

The dataset is not authorized to the current user

DATASET_NOT_ACTIVEstr

The dataset is inactive

MULTIPLE_ATTACHMENT_TYPE_CODEstr

Multiple attachments of the same type are not allowed

MULTIPLE_CODE_ATTACHMENTstr

Multiple attachments (commune, maille, departement) with the same code were given.

INVALID_DATEstr

The date is not valid

INVALID_UUIDstr

The uuid is not valid

INVALID_INTEGERstr

The integer is not valid

INVALID_NUMERICstr

The numeric is not valid

INVALID_WKTstr

The WKT string is not valid

INVALID_GEOMETRYstr

The geometry is not valid

INVALID_BOOLstr

The boolean is not valid

INVALID_ATTACHMENT_CODEstr

The code given does not exists in the desitination referential

INVALID_CHAR_LENGTHstr

The character length is not valid

DATE_MIN_TOO_HIGHstr

The date min is too high

DATE_MAX_TOO_LOWstr

The date max is too low

DATE_MAX_TOO_HIGHstr

The date max is too high

DATE_MIN_TOO_LOWstr

The date min is too low

ALTI_MIN_SUP_ALTI_MAXstr

The altitude min is superior to the altitude max

DATE_MIN_SUP_DATE_MAXstr

The date min is superior to the date max

DEPTH_MIN_SUP_ALTI_MAXstr

The depth min is superior to the altitude max

ORPHAN_ROWstr

The row could not be attached to an other entity # FIXME: clarify

DUPLICATE_ROWSstr

One rows appears more than once

DUPLICATE_UUIDstr

A uuid value is duplicated

EXISTING_UUID: str

A uuid value already exists in the destination table

SKIP_EXISTING_UUID: str

A uuid value already exists in the destination table and should be skipped

MISSING_VALUEstr

A required value is missing (see mandatory column in gn_imports.bib_fields table)

MISSING_GEOMstr

The geometry is missing

GEOMETRY_OUTSIDEstr

The geometry is outside the polygon defined by ID_AREA_RESTRICTION in the configuration

NO_GEOMstr

No geometry given (wherever WKT or latitude/longitude)

GEOMETRY_OUT_OF_BOXstr

The geometry is outside of a bounding box

ERRONEOUS_PARENT_ENTITYstr

The parent entity is not valid

NO_PARENT_ENTITYstr

The parent entity is not found

DUPLICATE_ENTITY_SOURCE_PKstr

The entity source primary key is duplicated

COUNT_MIN_SUP_COUNT_MAXstr

The count min is superior to the count max

INVALID_NOMENCLATUREstr

The nomenclature is invalid

INVALID_EXISTING_PROOF_VALUEstr

The existing proof value is invalid

CONDITIONAL_MANDATORY_FIELD_ERRORstr

Some conditional mandatory fields are missing #FIXME: clarify

INVALID_NOMENCLATURE_WARNINGstr

The nomenclature is invalid

UNKNOWN_ERRORstr

An unknown error occurred

INVALID_STATUT_SOURCE_VALUEstr

The statut source value is invalid

CONDITIONAL_INVALID_DATAstr

The conditional data is invalid

INVALID_URL_PROOFstr

The url proof is invalid

ROW_HAVE_TOO_MUCH_COLUMNstr

A row have too much column

ROW_HAVE_LESS_COLUMNstr

A row have less column

EMPTY_ROWstr

A row is empty

HEADER_SAME_COLUMN_NAMEstr

The header have same column name

EMPTY_FILEstr

The file is empty

NO_FILE_SENDEDstr

No file was sent

ERROR_WHILE_LOADING_FILEstr

An error occurred while loading the file

FILE_FORMAT_ERRORstr

The file format is not valid

FILE_EXTENSION_ERRORstr

The file extension is not valid

FILE_OVERSIZEstr

The file is too big

FILE_NAME_TOO_LONGstr

The file name is too long

FILE_WITH_NO_DATAstr

The file have no data

INCOHERENT_DATAstr

An entity data is different in multiple rows

CD_HAB_NOT_FOUNDstr

The habitat code is not found

CD_NOM_NOT_FOUNDstr

The cd_nom is not found in the instance TaxRef

DATASET_NOT_FOUND = 'DATASET_NOT_FOUND'

DATASET_NOT_AUTHORIZED = 'DATASET_NOT_AUTHORIZED'

DATASET_NOT_ACTIVE = 'DATASET_NOT_ACTIVE'

MULTIPLE_ATTACHMENT_TYPE_CODE = 'MULTIPLE_ATTACHMENT_TYPE_CODE'

MULTIPLE_CODE_ATTACHMENT = 'MULTIPLE_CODE_ATTACHMENT'

INVALID_DATE = 'INVALID_DATE'

INVALID_UUID = 'INVALID_UUID'

INVALID_INTEGER = 'INVALID_INTEGER'

INVALID_NUMERIC = 'INVALID_NUMERIC'

INVALID_WKT = 'INVALID_WKT'

INVALID_GEOMETRY = 'INVALID_GEOMETRY'

INVALID_BOOL = 'INVALID_BOOL'

INVALID_ATTACHMENT_CODE = 'INVALID_ATTACHMENT_CODE'

INVALID_CHAR_LENGTH = 'INVALID_CHAR_LENGTH'

DATE_MIN_TOO_HIGH = 'DATE_MIN_TOO_HIGH'

DATE_MAX_TOO_LOW = 'DATE_MAX_TOO_LOW'

DATE_MAX_TOO_HIGH = 'DATE_MAX_TOO_HIGH'

DATE_MIN_TOO_LOW = 'DATE_MIN_TOO_LOW'

DATE_MIN_SUP_DATE_MAX = 'DATE_MIN_SUP_DATE_MAX'

DEPTH_MIN_SUP_ALTI_MAX = 'DEPTH_MIN_SUP_ALTI_MAX'

ALTI_MIN_SUP_ALTI_MAX = 'ALTI_MIN_SUP_ALTI_MAX'

ORPHAN_ROW = 'ORPHAN_ROW'

DUPLICATE_ROWS = 'DUPLICATE_ROWS'

DUPLICATE_UUID = 'DUPLICATE_UUID'

EXISTING_UUID = 'EXISTING_UUID'

SKIP_EXISTING_UUID = 'SKIP_EXISTING_UUID'

MISSING_VALUE = 'MISSING_VALUE'

MISSING_GEOM = 'MISSING_GEOM'

GEOMETRY_OUTSIDE = 'GEOMETRY_OUTSIDE'

NO_GEOM = 'NO-GEOM'

GEOMETRY_OUT_OF_BOX = 'GEOMETRY_OUT_OF_BOX'

ERRONEOUS_PARENT_ENTITY = 'ERRONEOUS_PARENT_ENTITY'

NO_PARENT_ENTITY = 'NO_PARENT_ENTITY'

DUPLICATE_ENTITY_SOURCE_PK = 'DUPLICATE_ENTITY_SOURCE_PK'

COUNT_MIN_SUP_COUNT_MAX = 'COUNT_MIN_SUP_COUNT_MAX'

INVALID_NOMENCLATURE = 'INVALID_NOMENCLATURE'

INVALID_EXISTING_PROOF_VALUE = 'INVALID_EXISTING_PROOF_VALUE'

INVALID_NOMENCLATURE_WARNING = 'INVALID_NOMENCLATURE_WARNING'

CONDITIONAL_MANDATORY_FIELD_ERROR = 'CONDITIONAL_MANDATORY_FIELD_ERROR'

UNKNOWN_ERROR = 'UNKNOWN_ERROR'

INVALID_STATUT_SOURCE_VALUE = 'INVALID_STATUT_SOURCE_VALUE'

CONDITIONAL_INVALID_DATA = 'CONDITIONAL_INVALID_DATA'

INVALID_URL_PROOF = 'INVALID_URL_PROOF'

ROW_HAVE_TOO_MUCH_COLUMN = 'ROW_HAVE_TOO_MUCH_COLUMN'

ROW_HAVE_LESS_COLUMN = 'ROW_HAVE_LESS_COLUMN'

EMPTY_ROW = 'EMPTY_ROW'

HEADER_SAME_COLUMN_NAME = 'HEADER_SAME_COLUMN_NAME'

EMPTY_FILE = 'EMPTY_FILE'

NO_FILE_SENDED = 'NO_FILE_SENDED'

ERROR_WHILE_LOADING_FILE = 'ERROR_WHILE_LOADING_FILE'

FILE_FORMAT_ERROR = 'FILE_FORMAT_ERROR'

FILE_EXTENSION_ERROR = 'FILE_EXTENSION_ERROR'

FILE_OVERSIZE = 'FILE_OVERSIZE'

FILE_NAME_TOO_LONG = 'FILE_NAME_TOO_LONG'

FILE_WITH_NO_DATA = 'FILE_WITH_NO_DATA'

INCOHERENT_DATA = 'INCOHERENT_DATA'

CD_HAB_NOT_FOUND = 'CD_HAB_NOT_FOUND'

CD_NOM_NOT_FOUND = 'CD_NOM_NOT_FOUND'

class geonature.core.imports.checks.sql.BibFields

Bases: geonature.utils.env.db.Model

__tablename__ = 'bib_fields'

__table_args__

id_field

id_destination

destination

name_field

source_field

dest_field

fr_label

eng_label

type_field

type_field_params

mandatory

autogenerated

mnemonique

nomenclature_type

display

multi

optional_conditions

mandatory_conditions

entities

property source_column

property dest_column

__repr__()

class geonature.core.imports.checks.sql.Entity

Bases: geonature.utils.env.db.Model

__tablename__ = 'bib_entities'

__table_args__

id_entity

id_destination

destination

code

label

order

validity_column

destination_table_schema

destination_table_name

id_unique_column

id_uuid_column

id_parent

parent

childs

fields

unique_column

uuid_column

id_object

object

get_destination_table()

__repr__()

class geonature.core.imports.checks.sql.TImports

Bases: InstancePermissionMixin, geonature.utils.env.db.Model

__tablename__ = 't_imports'

__table_args__

AVAILABLE_ENCODINGS

AVAILABLE_FORMATS = ['csv', 'geojson']

AVAILABLE_SEPARATORS = [',', ';']

id_import

id_destination

destination

format_source_file

srid

separator

detected_separator

encoding

detected_encoding

full_file_name

date_create_import

date_update_import

date_end_import

source_count

erroneous_rows

statistics

date_min_data

date_max_data

uuid_autogenerated

altitude_autogenerated

authors

loaded

processed

source_file

columns

fieldmapping

contentmapping

observermapping

task_id

errors

property cruved

errors_count

property task_progress

has_instance_permission(scope, user=None, action_code='C')

static filter_by_scope(scope, user=None, **kwargs)

as_dict(import_as_dict)

geonature.core.imports.checks.sql.get_duplicates_query(imprt, dest_field, whereclause=sa.true())

geonature.core.imports.checks.sql.report_erroneous_rows(imprt, entity, error_type, error_column, whereclause, error_comment=None, level_validity_mapping={'ERROR': False})

Report erroneous rows in a transient table and update validity of rows based on error level.

This function reports errors found in imported data based on a WHERE clause, and updates the validity column of affected rows if the error level is specified in level_validity_mapping. By default, errors with the level « ERROR » mark the row as invalid.

Parameters

imprtTImport

The current import object.

entityEntity

The entity associated with the error. Must be defined if the error level is « ERROR ».

error_typestr

Type of error to report. Must correspond to a record in ImportUserErrorType.

error_columnstr

Name of the column where the error is detected. Can be mapped via imprt.fieldmapping.

whereclausesqlalchemy.sql.elements.ClauseElement

SQL clause defining the rows affected by the error.

error_commentstr, optional

Optional comment to include extra explanation to describe the error in the current import context.

level_validity_mappingdict, optional

Dictionary mapping error levels to validity values: - If the level is not in the dictionary, the row validity remains unchanged. - If the level is present, validity is set according to the associated value:

• False: The row is marked as erroneous.

• None: The row is marked as should not be imported.

By default, only the level « ERROR » is mapped to False.

Raises

AssertionError

If entity is not defined for an error of level « ERROR ».

Examples

>>> # Example usage for reporting an "ERROR" level error
>>> report_erroneous_rows(
...     imprt=my_import,
...     entity=my_entity,
...     error_type="MISSING_VALUE",
...     error_column="customer_name",
...     whereclause=(transient_table.c.customer_name == None),
...     error_comment="Customer name missing",
... )

geonature.core.imports.checks.sql.check_referential(imprt: geonature.core.imports.models.TImports, entity: geonature.core.imports.models.Entity, field: geonature.core.imports.models.BibFields, reference_field: sqlalchemy.Column, error_type: str, reference_table: sqlalchemy.Table | None = None) → None

Check the referential integrity of a column in the transient table.

Parameters

imprtTImports

The import to check.

entityEntity

The entity to check.

fieldBibFields

The field to check.

reference_fieldBibFields

The reference field to check.

error_typestr

The type of error encountered.

reference_tableOptional[sa.Table], optional

The reference table to check. If not provided, it will be inferred from the reference_field.

geonature.core.imports.checks.sql.check_cd_nom(imprt: geonature.core.imports.models.TImports, entity: geonature.core.imports.models.Entity, field: geonature.core.imports.models.BibFields, list_id: int | None = None) → None

Check the existence of a cd_nom in the Taxref referential.

Parameters

imprtTImports

The import to check.

entityEntity

The entity to check.

fieldBibFields

The field to check.

list_idOptional[int], optional

The list to filter on, by default None.

geonature.core.imports.checks.sql.check_cd_hab(imprt: geonature.core.imports.models.TImports, entity: geonature.core.imports.models.Entity, field: geonature.core.imports.models.BibFields) → None

Check the existence of a cd_hab in the Habref referential.

Parameters

imprtTImports

The import to check.

entityEntity

The entity to check.

fieldBibFields

The field to check.

geonature.core.imports.checks.sql.generate_altitudes(imprt: geonature.core.imports.models.TImports, geom_local_field: geonature.core.imports.models.BibFields, alt_min_field: geonature.core.imports.models.BibFields, alt_max_field: geonature.core.imports.models.BibFields) → None

Generate the altitudes based on geomatries, and given altitues in an import.

Parameters

imprtTImports

The import to generate altitudes for.

geom_local_fieldBibFields

The field representing the geometry in the destination import’s transient table.

alt_min_fieldBibFields

The field representing the minimum altitude in the destination import’s transient table.

alt_max_fieldBibFields

The field representing the maximum altitude in the destination import’s transient table.

geonature.core.imports.checks.sql.check_duplicate_uuid(imprt: geonature.core.imports.models.TImports, entity: geonature.core.imports.models.Entity, uuid_field: geonature.core.imports.models.BibFields)

Check if there is already a record with the same uuid in the transient table. Include an error in the report for each entry with a uuid dupplicated.

Parameters

imprtImport

The import to check.

entityEntity

The entity to check.

uuid_fieldBibFields

The field to check.

geonature.core.imports.checks.sql.check_existing_uuid(imprt: geonature.core.imports.models.TImports, entity: geonature.core.imports.models.Entity, uuid_field: geonature.core.imports.models.BibFields, id_dataset_field: geonature.core.imports.models.BibFields = None, skip=False)

Check if there is already a record with the same uuid in the destination table. Include an error in the report for each existing uuid in the destination table. Parameters ———- imprt : Import

The import to check.

entityEntity

The entity to check.

uuid_fieldBibFields

The field to check

id_dataset_fieldBibFields

if defnied, the uuid existence is checked for the given dataset. Otherwise, it is checked globally

skip: Boolean

Raise SKIP_EXISTING_UUID instead of EXISTING_UUID and set row validity to None (do not import)

geonature.core.imports.checks.sql.generate_missing_uuid_for_id_origin(imprt: geonature.core.imports.models.TImports, uuid_field: geonature.core.imports.models.BibFields, id_origin_field: geonature.core.imports.models.BibFields)

Update records in the transient table where the uuid is None with a new UUID. Generate UUID in transient table when there are no UUID yet, but there are a id_origin. Ensure rows with same id_origin get the same UUID.

Parameters

imprtTImports

The import to check.

uuid_fieldBibFields

The field to check.

id_origin_fieldBibFields

Field used to generate the UUID

geonature.core.imports.checks.sql.generate_missing_uuid(imprt: geonature.core.imports.models.TImports, entity: geonature.core.imports.models.Entity, uuid_field: geonature.core.imports.models.BibFields, whereclause: Any = None)

Update records in the transient table where the UUID is None with a new UUID.

Parameters

imprtTImports

The import to check.

entityEntity

The entity to check.

uuid_fieldBibFields

The field to check.

geonature.core.imports.checks.sql.check_duplicate_source_pk(imprt: geonature.core.imports.models.TImports, entity: geonature.core.imports.models.Entity, field: geonature.core.imports.models.BibFields) → None

Check for duplicate source primary keys in the transient table of an import.

Parameters

imprtTImports

The import to check.

entityEntity

The entity to check.

fieldBibFields

The field to check.

geonature.core.imports.checks.sql.check_dates(imprt: geonature.core.imports.models.TImports, entity: geonature.core.imports.models.Entity, date_min_field: geonature.core.imports.models.BibFields = None, date_max_field: geonature.core.imports.models.BibFields = None) → None

Check the validity of dates in the transient table of an import.

Parameters

imprtTImports

The import to check.

entityTEntity

The entity to check.

date_min_fieldBibFields, optional

The field representing the minimum date.

date_max_fieldBibFields, optional

The field representing the maximum date.

geonature.core.imports.checks.sql.check_altitudes(imprt: geonature.core.imports.models.TImports, entity: geonature.core.imports.models.Entity, alti_min_field: geonature.core.imports.models.BibFields = None, alti_max_field: geonature.core.imports.models.BibFields = None) → None

Check the validity of altitudes in the transient table of an import.

Parameters

imprtTImports

The import to check.

entityTEntity

The entity to check.

alti_min_fieldBibFields, optional

The field representing the minimum altitude.

alti_max_fieldBibFields, optional

The field representing the maximum altitude.

geonature.core.imports.checks.sql.check_depths(imprt: geonature.core.imports.models.TImports, entity: geonature.core.imports.models.Entity, depth_min_field: geonature.core.imports.models.BibFields = None, depth_max_field: geonature.core.imports.models.BibFields = None) → None

Check the validity of depths in the transient table of an import.

Parameters

imprtTImports

The import to check.

entityTEntity

The entity to check.

depth_min_fieldBibFields, optional

The field representing the minimum depth.

depth_max_fieldBibFields, optional

The field representing the maximum depth.

geonature.core.imports.checks.sql.check_digital_proof_urls(imprt, entity, digital_proof_field)

Checks for valid URLs in a given column of a transient table.

Parameters

imprtTImports

The import to check.

entityTEntity

The entity to check.

digital_proof_fieldTField

The field containing the URLs to check.

geonature.core.imports.checks.sql.check_entity_data_consistency(imprt, entity, fields, grouping_field)

Checks for rows with the same uuid, but different contents, in the same entity. Used mainely for parent entities. Parameters ———- imprt : TImports

The import to check.

entityEntity

The entity to check.

fieldsBibFields

The fields to check.

grouping_fieldBibFields

The field to group identical rows.

geonature.core.imports.checks.sql.disable_duplicated_rows(imprt, entity, fields, grouping_field)

When several rows have the same value in grouping field (typically UUID) field, first one is untouched but following rows have validity set to None (do not import).

geonature.core.imports.checks.sql.generate_entity_id(imprt: geonature.core.imports.models.TImports, entity: geonature.core.imports.models.Entity, schema_name: str, table_name: str, uuid_field_name: str, id_field_name: str) → None

Generate the id for each new valid entity

Parameters

imprtTImports

_description_

entityEntity

entity

geonature.core.imports.checks.sql.set_parent_id_from_line_no(imprt: geonature.core.imports.models.TImports, entity: geonature.core.imports.models.Entity, parent_line_no_field_name: str, parent_id_field_name: str) → None

Set the parent id of each entity in the transient table using the line_no of the corresponding parent.

Parameters

imprtTImports

The import object containing the destination.

entityEntity

entity

geonature.core.imports.checks.sql.set_id_parent_from_destination(imprt: geonature.core.imports.models.TImports, parent_entity: geonature.core.imports.models.Entity, entity: geonature.core.imports.models.Entity, id_field: geonature.core.imports.models.BibFields, fields: List[geonature.core.imports.models.BibFields]) → None

Complete the id_parent column in the transient table of an import when the parent already exists in the destination table.

Parameters

imprtTImports

The import to update.

parent_entityEntity

The entity of the parent.

entityEntity

The current entity.

id_fieldBibFields

The field containing the id of the parent.

fieldsList[BibFields]

The fields to use for matching the child with its parent in the destination table.

geonature.core.imports.checks.sql.set_parent_line_no(imprt: geonature.core.imports.models.TImports, parent_entity: geonature.core.imports.models.Entity, entity: geonature.core.imports.models.Entity, parent_line_no: geonature.core.imports.models.BibFields, fields: List[geonature.core.imports.models.BibFields]) → None

Set parent_line_no on child entities when: - no parent entity on same line - parent entity is valid - looking for parent entity through each given field in fields

Parameters

imprtTImports

The import to update.

parent_entityEntity

The entity of the parent.

entityEntity

The current entity.

id_parentBibFields

The field containing the id of the parent.

parent_line_noBibFields

The field containing the line number of the parent.

fieldsList[BibFields]

The fields to use for matching the child with its parent in the destination table.

geonature.core.imports.checks.sql.check_no_parent_entity(imprt: geonature.core.imports.models.TImports, parent_entity: geonature.core.imports.models.Entity, entity: geonature.core.imports.models.Entity, id_parent: geonature.core.imports.models.BibFields, parent_line_no: geonature.core.imports.models.BibFields) → None

Station may be referenced: - on the same line (station_validity is not None) - by id_parent (parent already exists in destination) - by parent_line_no (new parent from another line of the imported file - see set_parent_line_no)

Parameters

imprtTImports

The import to check.

parent_entityEntity

The entity of the parent.

entityEntity

The current entity.

id_parentBibFields

The field containing the id of the parent.

parent_line_noBibFields

The field containing the line number of the parent.

geonature.core.imports.checks.sql.check_erroneous_parent_entities(imprt: geonature.core.imports.models.TImports, parent_entity: geonature.core.imports.models.Entity, entity: geonature.core.imports.models.Entity, parent_line_no: geonature.core.imports.models.BibFields) → None

Check for erroneous (not valid) parent entities in the transient table of an import.

Parameters

imprtTImports

The import to check.

parent_entityEntity

The entity of the parent.

entityEntity

The current entity.

parent_line_noBibFields

The field containing the line number of the parent.

Notes

# Note: if child entity reference parent entity by id_parent, this means the parent # entity is already in destination table so obviously valid.

The error codes are:

• ERRONEOUS_PARENT_ENTITY: the parent on the same line is not valid.