Integrations
Integrations
Breadcrumbs

File System - File Metadata


Introduction

The purpose of this document is to explain which file types metadata can be extracted from, their peculiarities, and what information is extracted from each one.

The name indicated for each file is the name of the attribute that should exist in Anjana (name of the attribute_definition table) in the object templates from which information is to be extracted.

Path Separator

The path-separator, or extraction path separator, is a configurable parameter. By default, the character “/” is used, but if the data structures to be extracted contain it in their name, for example, a different character can be configured instead, in order to facilitate the correct extraction of those tables. To change it, it must be configured in kerno, tot, and in the plugins that will be used, so that it is the same in all of them. The properties to modify are:

  • In Kerno

    • anjana.tot.extraction.pathSeparator: “/”

  • In Tot

    • tot.extraction.pathSeparator: “/”

  • In plugins:

    • Specifically for PowerBI and Tableau: totplugin.pathSeparator: “/”

    • Any other plugin that is not PowerBI or Tableau: totplugin.connection.[<connectionName>].technology.pathSeparator: “/”

Any character is allowed except “:“ (due to technology limitations, if the character “\“ is configured in the YML, it must be entered as “\\“ in order to be used).

If the AWS IAM, AWS S3, and/or Azure Storage plugins are being used, these properties should not be given a value (neither in kerno, tot, nor in the plugins), and the default value should be left as is.

This separator is chosen at the time of installation and should not be changed given its importance for locating objects in the corresponding technology for data sampling and governance. If it is absolutely necessary to change it, consult with Anjana Support.

File Types

CSV

CSV type files are identified by their “.csv” extension. Each column will be interpreted as a dataset_field, for which the following information will be populated:

  • name with the field name

  • physical_name with the field name

  • fieldDataType with the data type defined for the field (can be boolean, number, string, or date)

  • position indicating the position occupied by the field

The separators supported for CSV extraction are comma (,), semicolon (;), and tab.

AVRO

AVRO type files are identified by their “.avro” extension. These files can be single or partitioned.

Each column will be interpreted as a dataset_field, for which the following information will be populated:

  • name with the field value

  • physical_name with the field name

  • defaultValue with the default value defined for the field

  • fieldDataType with the data type defined for the field (can be record, enum, array, map, union, fixed, string, bytes, int, long, float, double, boolean, or null)

  • position indicating the position occupied by the field

  • description with the field's description

  • alias the aliases the field has

In addition to these values present in every field of an avro file, extra properties can be added; all properties included will be collected and extracted.

EXCEL

Excel type files are identified by their “.xls” and “.xlsx” extensions. Each column will be interpreted as a dataset_field, for which the following information will be populated:

  • name with the field value

  • physical_name with the field name

  • fieldDataType with the data type defined for the field (can be string, boolean, number)

  • position indicating the position occupied by the field

  • description with the field's description

PARQUET

Parquet type files are identified by their “.parquet” extension. These files can be single or partitioned.

Each column will be interpreted as a dataset_field, for which the following information will be populated:

  • name with the field value

  • physical_name with the field name

  • fieldDataType with the data type defined for the field (can be int64, int32, boolean, binary, float, double, int96, or fixed_len_type_array)

  • position indicating the position occupied by the field

  • nullable indicating whether the field is nullable

  • length indicating the field's length

Only fields belonging to primitive types will be extracted.

Organization Standards

HADOOP

Allowed directory types:

With a single file at the end

/folder_1_lvl1
    /folder_1_1_lvl2
        file.extension

With multiple parts of the same file at the end

/folder_1_lvl1
    /date=feb
        part.000001.name.parquet
        part.000002.name.parquet
    /date=march
        part.000003.name.parquet
        part.000004.name.parquet

All files contained within the same directory that have a name with the same number of characters will be counted as parts of the same file.

Allowed file types:

  • Parquet

  • Avro

  • CSV

  • Excel

Naming convention
Files

Files must follow a naming convention of part.000000.name.extension for partitioned files and name.extension for complete files. Replacing the ‘0’s with the desired value to indicate that it is part X of a file; that is, if a file had 2 parts, there would be part.000001.name.extension and part.000002.name.extension, for example.

Directories

Directories must follow a pattern that indicates the level of each directory, for example:

/folder_1_lvl1
    /folder_1_1_lvl2
        part.000001.name.parquet
        part.000002.name.parquet
    /folder_1_2_lvl2
        /folder_1_2_1_lvl3
            part.000001.name.parquet
            part.000002.name.parquet

Delta Lake

Allowed directory types:

With a single file at the end

/folder_1_lvl1
    /folder_1_1_lvl2
        /_delta_log
        file.parquet

Note that in this format there is a folder specific to DeltaLake whose content is ignored.

With multiple parts of the same file at the end

/folder_1_lvl1
    /folder_1_1_lvl2
        /_delta_log
            000000.json
        part.000001.name.parquet
        part.000002.name.parquet

  • All files contained within the same directory that have a name with the same number of characters will be counted as parts of the same file.

  • Note that in this format there is a folder specific to DeltaLake whose content is ignored.

With multiple parts of the same file at the end and with partitions

/folder_1_lvl1
    /folder_1_1_lvl2
        /_delta_log
            000000.json
        /partition
            part.000001.name.parquet
            part.000002.name.parquet
        part.000001.name.parquet
        part.000002.name.parquet

  • All files contained within the same directory that have a name with the same number of characters will be counted as parts of the same file.

  • Note that in this format there is a folder specific to DeltaLake whose content is ignored.

Allowed file types:

Parquet

Naming convention
Files

Files must follow a naming convention of part.000000.name.parquet for partitioned files and name.parquet for complete files. Replacing the ‘0’s with the desired value to indicate that it is part X of a file; that is, if a file had 2 parts, there would be part.000001.name.parquet and part.000002.name.parquet, for example.

Directories

Directories must follow a pattern that indicates the level of each directory, for example:

/folder_1_lvl1
    /folder_1_1_lvl2
        /_delta_log
            000000.json
        part.000001.name.parquet
        part.000002.name.parquet
    /folder_1_2_lvl2
        /folder_1_2_1_lvl3
            /_delta_log
                000000.json
            part.000001.name.parquet
            part.000002.name.parquet

They must also include the _delta_log folder inside each directory with files, as well as another folder called partition with files.