XML files and custom Input Formats

Overview

In Supplier Data Manager (SDM), custom XML input formats are predefined templates that specify how XML files should be processed. They allow configurable extraction of structured data based on XML paths and element relationships.

Custom XML input formats are managed in the SDM admin at Admin > Files > Input formats. For general Custom Input Format configuration, see Inputs.

Warning

Adding a custom input format for your expected XML files is mandatory before you can upload XML files.

What is XML?

XML (eXtensible Markup Language) is one of the file formats supported by Supplier Data Manager. While less commonly used than CSV or XLS, XML's flexible structure often requires more configuration to extract data properly. SDM's parsing system handles complex XML structures through configurable extraction rules that navigate hierarchical data, process attributes, and manage namespaces.

A typical product XML file looks like:

<?xml version="1.0" encoding="UTF-8"?>
<products>
  <product>
    <id>1</id>
    <name>Product One</name>
    <price currency="USD">19.99</price>
    <categories>
      <category>Electronics</category>
      <category>Gadgets</category>
    </categories>
  </product>
  <product>
    <id>2</id>
    <name>Product Two</name>
    <price currency="USD">29.99</price>
    <categories>
      <category>Home</category>
      <category>Kitchen</category>
    </categories>
  </product>
</products>

XML namespaces

XML namespaces help avoid element name conflicts when combining XML documents from different sources. They are defined using the xmlns attribute and represented as URIs.

Example of an XML document with namespaces:

<?xml version="1.0" encoding="UTF-8"?>
<ProductFeed xmlns="http://www.example.com/xsd/product/feed/0.2">
  <Products xmlns="http://www.example.com/xsd/product/0.3">
    <Product xmlns="http://www.example.com/xsd/product/0.3">
      <Id xmlns="http://www.example.com/xsd/product/0.3">1</Id>
      <Name xmlns="http://www.example.com/xsd/product/0.3">Product One</Name>
      <Price xmlns="http://www.example.com/xsd/product/0.3" currency="USD">19.99</Price>
    </Product>
  </Products>
</ProductFeed>

Handling namespaces in SDM

When SDM parses XML files with namespaces, you have two options:

1. Remove namespaces: Treats all elements as if they had no namespace, simplifying the parsing process. This works well for most data extraction scenarios.
2. Preserve namespaces (default): Keeps the namespace information intact. Use this when dealing with elements that share the same local name but belong to different namespaces.

The remove_ns option

Control namespace handling using the remove_ns parameter in your XML format configuration:

{
  "sources": [
    {
      "root_paths": [["ProductFeed", "Products", "Product"]],
      "remove_ns": true,
      "columns": [...]
    }
  ]
}

Default value: false (namespaces are preserved).

When to use remove_ns: true

• For most simple data extraction scenarios where namespace information isn't relevant
• When you want to use simpler element paths without namespace prefixes

When to use remove_ns: false (default)

• When working with elements that share the same local name but have different namespaces
• When namespace information is semantically important

Example

Consider this XML with namespaces:

<ns1:product xmlns:ns1="http://example.com/ns1" xmlns:ns2="http://example.com/ns2">
  <ns1:id>1</ns1:id>
  <ns1:name>Product One</ns1:name>
  <ns2:price currency="USD">19.99</ns2:price>
</ns1:product>

With remove_ns: true, SDM treats this as:

<product>
  <id>1</id>
  <name>Product One</name>
  <price currency="USD">19.99</price>
</product>

This lets you extract data using simple paths like ["product", "id"].

With remove_ns: false (default), paths must include namespace prefixes — for example, ["ns1:product", "ns1:id"].

How custom XML input formats work

A custom XML input format in Supplier Data Manager lets you define:

1. Where to find product listings in the XML structure (root paths)
2. Which XML elements to extract as columns
3. How to name those columns
4. How to handle XML attributes

Workflow

The diagram above shows the XML parsing workflow in Supplier Data Manager: the uploaded file is parsed, checked against any defined custom XML input format, and extracted into a structured data frame. If no matching format is found, SDM uses its default XML parsing rules.

Configuration options

In Supplier Data Manager, a custom XML input format is a JSON object with a sources array. Each source defines a distinct extraction pattern. SDM tries each source in order and uses the first one that matches the uploaded file's structure.

Sources

A custom XML input format can define multiple sources, each representing a different extraction pattern. For each source, you can configure the following options.

Root paths

Root paths define where to find product listings in the XML structure. Each root path is a sequence of XML tag names leading to the elements that contain product data.

Info

The topmost parent element of the XML file (the document root) does not need to be included in root paths. SDM automatically starts searching from the document root.

Example configuration:

"root_paths": [["products", "product"]]

This matches all <product> elements under the <products> element:

<?xml version="1.0" encoding="UTF-8"?>
<products_feed>
  <products>
    <product>
      <id>1</id>
      <name>Product One</name>
    </product>
    <product>
      <id>2</id>
      <name>Product Two</name>
    </product>
  </products>
</products_feed>

products_feed (the document root) is not included in the root path, but SDM still correctly locates the <product> elements. Each <product> element is processed as a separate row in the resulting data.

Warning

XML data extraction in SDM focuses on repeating elements at a single structural level. Cross-referencing data from parent elements, sibling nodes, or different document sections is not supported.

Exclude paths

Specify paths to exclude from parsing to ignore certain elements within your product listings.

Example configuration:

"exclude": [["product", "internal_data"]]

With this XML:

<products>
  <product>
    <id>1</id>
    <name>Product One</name>
    <internal_data>
      <cost>10.00</cost>
      <supplier_id>S123</supplier_id>
    </internal_data>
    <price>19.99</price>
  </product>
</products>

SDM extracts id, name, and price, and ignores the entire internal_data element and its children (cost and supplier_id).

Name mode

The name mode determines how column names are generated from XML paths.

• join: Joins all tags in the path using a separator (default separator: /)
• last: Uses only the last tag in the path as the column name

Default value: join

Example configuration:

"name_mode": "join"

With this XML:

<product>
  <details>
    <color>Red</color>
    <size>Large</size>
  </details>
</product>

name_mode: "join" creates columns:

• details/color with value "Red"
• details/size with value "Large"

name_mode: "last" creates columns:

• color with value "Red"
• size with value "Large"

Name separator

When using name_mode: "join", this option defines the separator between tags. The separator can be up to 5 characters long.

Default value: /

Example configuration:

"name_mode": "join",
"name_separator": "_"

With the same XML as above, this creates columns:

• details_color with value "Red"
• details_size with value "Large"

Name with attributes

Determines whether XML attributes are included in column names.

Default value: false

Example configuration:

"name_with_attributes": true

With this XML:

<product>
  <price currency="USD" tax="included">19.99</price>
  <weight unit="kg">1.5</weight>
</product>

This creates columns that include attributes in their names:

• price_currency:USD_tax:included with value "19.99"
• weight_unit:kg with value "1.5"

Attributes template

When name_with_attributes is true, this template controls how attributes are formatted in column names. Use {name} and {value} as placeholders.

Default value: _{name}:{value}

Example configuration:

"name_with_attributes": true,
"attributes_template": "-{name}({value})"

With the same XML as above, this creates:

• price-currency(USD)-tax(included) with value "19.99"

Columns

For each source, you define which columns to extract. Each column entry supports the following options.

Path

The path to the XML element to extract, relative to the root path.

Example configuration:

"path": ["name"]

With this XML:

<product>
  <id>1</id>
  <name>Product One</name>
  <price>19.99</price>
</product>

This extracts the content of the <name> element, creating a column with value "Product One".

To extract all elements and rely on exclude to filter out unwanted ones, use null as the path value:

"columns": [{"path": [null]}]

XPath predicates for attribute filtering

In Supplier Data Manager, you can use XPath predicates in path elements to filter by XML attributes. This is particularly useful for multilingual XML files.

Supported predicates:

• element[not(@attribute)] — Match elements without a specific attribute
• element[@attribute='value'] — Match elements with a specific attribute value
• element[@attribute] — Match elements that have a specific attribute (any value)

Example: multilingual XML

Consider this XML with default and English translations:

<RECIPE>
  <TITLE>Spaghetti alla Carbonara</TITLE>
  <TITLE lang="eng">Pasta Carbonara</TITLE>
  <DESCRIPTION>Plat italien à base d'œufs, de pecorino et de pancetta.</DESCRIPTION>
  <DESCRIPTION lang="eng">Italian dish made with eggs, pecorino, and pancetta.</DESCRIPTION>
  <COOKTIME>20</COOKTIME>
</RECIPE>

You can extract specific language versions:

"columns": [
  {
    "path": ["RECIPE", "TITLE[not(@lang)]"],
    "name": "title_default"
  },
  {
    "path": ["RECIPE", "TITLE[@lang='eng']"],
    "name": "title_english"
  },
  {
    "path": ["RECIPE", "DESCRIPTION[not(@lang)]"],
    "name": "description_default"
  },
  {
    "path": ["RECIPE", "DESCRIPTION[@lang='eng']"],
    "name": "description_english"
  }
]

This creates four columns:

• title_default: "Spaghetti alla Carbonara" (no lang attribute)
• title_english: "Pasta Carbonara" (lang="eng")
• description_default: "Plat italien à base d'œufs, de pecorino et de pancetta."
• description_english: "Italian dish made with eggs, pecorino, and pancetta."

Using XPath predicates with exclude paths

XPath predicates also work in the exclude array to filter out specific elements based on their attributes. This is useful when you want to exclude certain language versions while keeping others.

Example: excluding English-language elements while keeping German:

{
  "sources": [
    {
      "root_paths": [["CATALOG", "PRODUCT"]],
      "exclude": [
        ["PRODUCT_DETAILS", "DESCRIPTION[@lang='eng']"],
        ["PRODUCT_DETAILS", "KEYWORD[@lang='eng']"],
        ["PRODUCT_FEATURES", "FEATURE", "FNAME[@lang='eng']"],
        ["PRODUCT_FEATURES", "FEATURE", "FVALUE[@lang='eng']"]
      ],
      "columns": [{"path": [null]}]
    }
  ]
}

With this XML:

<PRODUCT>
  <PRODUCT_DETAILS>
    <DESCRIPTION>Deutsche Beschreibung</DESCRIPTION>
    <DESCRIPTION lang="deu">Deutsche Beschreibung</DESCRIPTION>
    <DESCRIPTION lang="eng">English Description</DESCRIPTION>
    <KEYWORD lang="deu">Schlüsselwort</KEYWORD>
    <KEYWORD lang="eng">Keyword</KEYWORD>
  </PRODUCT_DETAILS>
  <PRODUCT_FEATURES>
    <FEATURE>
      <FNAME>Eigenschaft</FNAME>
      <FNAME lang="deu">Eigenschaft</FNAME>
      <FNAME lang="eng">Property</FNAME>
      <FVALUE>Wert</FVALUE>
      <FVALUE lang="eng">Value</FVALUE>
    </FEATURE>
  </PRODUCT_FEATURES>
</PRODUCT>

SDM excludes all elements with lang="eng", keeping the default (no lang attribute) and lang="deu" versions:

• "Deutsche Beschreibung" (default and lang="deu")
• "Schlüsselwort" (lang="deu")
• "Eigenschaft" (default and lang="deu")
• "Wert" (default)

Warning

XPath predicates are slightly slower than simple tag matching. Use them only where needed for best performance.

Name

An optional custom name for the column. If not specified, the name is generated from the path and name mode settings.

Example configuration:

"path": ["name"],
"name": "product_name"

Instead of using "name" as the column name, this uses "product_name" for the column containing "Product One".

Name from / Value from

These options dynamically extract column names and values from the XML. Use them when the column names themselves are stored inside the XML structure.

Example configuration:

"path": ["attributes", "attribute"],
"name_from": ["name"],
"value_from": ["value"]

With this XML:

<product>
  <id>1</id>
  <attributes>
    <attribute>
      <name>Color</name>
      <value>Red</value>
    </attribute>
    <attribute>
      <name>Size</name>
      <value>Large</value>
    </attribute>
    <attribute>
      <name>Material</name>
      <value>Cotton</value>
    </attribute>
  </attributes>
</product>

This dynamically creates three columns:

• Color with value "Red"
• Size with value "Large"
• Material with value "Cotton"

This is useful for variable attribute sets where you don't know all possible attributes in advance.

Column-level overrides

The name_mode, name_with_attributes, and attributes_template options can be set at the column level to override the source-level defaults for a specific column. This lets you apply different naming rules to individual columns within the same source.

Example: applying name_with_attributes to only the price column while keeping default settings for other columns:

"columns": [
  {
    "path": ["id"],
    "name": "product_id"
  },
  {
    "path": ["price"],
    "name_with_attributes": true
  }
]

Example configuration

Here is a complete example of a custom XML input format configuration in Supplier Data Manager:

{
  "name": "Product Catalog Format",
  "params": {
    "sources": [
      {
        "root_paths": [["catalog", "products", "product"]],
        "name_mode": "join",
        "name_separator": "_",
        "name_with_attributes": true,
        "attributes_template": "_{name}:{value}",
        "exclude": [["product", "internal"]],
        "remove_ns": true,
        "columns": [
          {
            "path": ["id"],
            "name": "product_id"
          },
          {
            "path": ["name"],
            "name": "product_name"
          },
          {
            "path": ["price"],
            "name_with_attributes": true
          },
          {
            "path": ["attributes", "attribute"],
            "name_from": ["name"],
            "value_from": ["value"]
          }
        ]
      }
    ]
  }
}

Processing flow

The diagram above shows the processing flow: the user uploads an XML file via the API, the XML parser checks whether a matching custom input format exists, extracts data according to the format (or applies default rules if no format matches), converts the result to a structured data frame, and returns the response.

Best practices

1. Start with root paths: Correctly identify where your product data is located in the XML structure.
2. Use exclude paths to ignore irrelevant or sensitive data rather than listing every column explicitly.
3. Choose the appropriate name mode:
4. Use join when you want to preserve the full path hierarchy in column names.
5. Use last when you only need the leaf element name.
6. Use custom column names for clarity and consistency, especially when integrating with other systems.
7. Handle attributes appropriately: Use name_with_attributes and attributes_template to include XML attribute data in extracted columns.
8. Test with sample files: Always test your format configuration with representative files before deploying.
9. Consider namespace handling: Use the remove_ns parameter based on whether your XML uses semantically distinct namespaces.

Troubleshooting

Common issues

1. No data extracted: Check that your root paths correctly identify the product elements in the XML.
2. Missing columns: Verify that column paths are correct relative to the root path.
3. Encoding issues: If you encounter encoding problems, specify the encoding explicitly when uploading the file.
4. Performance with large files: Use more specific root paths to limit the data SDM processes.
5. Namespace issues: If you have trouble with namespaced XML:
6. Try setting remove_ns: true to simplify parsing.
7. If you need to preserve namespaces (the default), ensure your paths include the namespace prefixes.

Advanced usage

Dynamic column names

The name_from and value_from options in Supplier Data Manager extract column names dynamically from the XML. This is useful for attribute-value pair structures where column names are not known in advance.

Example XML:

<product>
  <id>123</id>
  <attributes>
    <attribute>
      <name>Color</name>
      <value>Red</value>
    </attribute>
    <attribute>
      <name>Size</name>
      <value>Large</value>
    </attribute>
  </attributes>
</product>

Configuration:

{
  "path": ["attributes", "attribute"],
  "name_from": ["name"],
  "value_from": ["value"]
}

This creates columns "Color" and "Size" with values "Red" and "Large".

Handling XML attributes

XML attributes can contain important data. Use the name_with_attributes and attributes_template options in SDM to include them in extracted columns.

Example XML:

<product>
  <price currency="USD">19.99</price>
</product>

With name_with_attributes: true and attributes_template: "_{name}:{value}", SDM creates a column named price_currency:USD with value "19.99".

Working with namespaced XML

When working with heavily namespaced XML (such as SOAP responses or industry-standard catalog formats), Supplier Data Manager supports two approaches:

Approach 1: Remove namespaces (simpler)

Set remove_ns: true to strip all namespace information:

{
  "sources": [
    {
      "root_paths": [["products", "product"]],
      "remove_ns": true,
      "columns": [...]
    }
  ]
}

This lets you use simple paths like ["product", "id"] regardless of the original namespace declarations.

Approach 2: Preserve namespaces (more precise, default)

Keep the default remove_ns: false to preserve namespace information:

{
  "sources": [
    {
      "root_paths": [["ns1:products", "ns1:product"]],
      "remove_ns": false,
      "columns": [{"path": ["ns1:id"]}, {"path": ["ns2:price"]}]
    }
  ]
}

Use this approach when:

• Elements share the same local name but belong to different namespaces
• You are validating against a schema
• The semantic meaning of the namespaces is important to your data

Limitations

For file size and format limits, see What product file formats and size are accepted?