Parameters

• job_id

  string

  Required

  Monitoring job ID

• task_id

  string

  Task ID to update (leave empty to create new task)

• task_name

  string

  Name for the task (required for new tasks)

• data_type

  string

  Optional semantic label for the data (e.g., 'News article', 'Job posting', 'Price'). Used to derive a Data Type when none provided.

• data_type_id

  string

  Optional: Explicit Data Type ID to use. When omitted, the tool infers a default type from steps/value type.

• steps

  array

  Required

  Extraction steps. Examples: - Single value: [{"method":"xPath","pattern":"//title"}] - Multiple items + transform: [{"method":"xPath_extract","pattern":"//div[@class=\"item\"]"},{"method":"json","pattern":null,"jsonConditions":[{"function":"transform","key":"{\"title\":\"${ @step | xPath : \'//h2\' }\"}"}]}]

  Array items

  • [item]

    object

    Nested fields

    • method

      string (xPath, xPath_extract, json, regex, gpt)

      xPath, xPath_extract, json, regex, or gpt

      Allowed values: xPath, xPath_extract, json, regex, gpt

    • pattern

      string

      Extraction pattern. For json, set to null when providing jsonConditions.

    • jsonConditions

      string

      Optional stringified conditions for json step (filters, transforms). Will be decoded/normalized.

• test_first

  boolean

  Test extraction before saving (recommended)

Parameters

• job_id

  string

  Required

  Monitoring job ID

• task_id

  string

  Required

  Task ID that will forward-download articles

• download_articles

  boolean

  Required

  Enable forward-download of links into articles

• story_channel_id

  string

  Story Channel ID (required when download_articles=true)

Parameters

• name

  string

  Required

  Name for the monitoring job

• url

  string

  Required

  URL to monitor

• data_model_name_or_id

  string

  Required

  Database name for storing raw results; typically an Entity name, such as Apple (creates if not exists)

• data_type_name_or_id

  string

  Optional: Name or ID of Data Type (e.g., "Blog article"). If name does not exist, it will be created.

• data_type_value_type

  string (Text, Json, Number, Date, Money, Boolean)

  Optional: Value type for new Data Type (defaults to Json).

  Allowed values: Text, Json, Number, Date, Money, Boolean

• download_articles

  boolean

  Optional: Enable forward download of extracted links into Stories (articles).

• story_channel_id

  string

  Optional: Story Channel ID to store downloaded articles in (required if download_articles=true).

• initial_task_name

  string

  Optional: Name of the initial task to create when download_articles is enabled.

• comment

  string

  Comment or description (optional)

Parameters

• source

  string

  Required

  URL (https://...), job_id (numeric), or "cached"/"same" to reuse last data

• pattern

  string

  Extraction pattern to test (optional). If not provided, shows data preview

• method

  string (xPath, xPath_extract, json, regex, gpt)

  Extraction method (auto-detected if not provided)

  Allowed values: xPath, xPath_extract, json, regex, gpt

• transform

  array

  Transformation mapping to apply after extraction (e.g., {"title": "@name", "price": "@cost"})

  Array items

  • [item]

    string

• previous_results

  array

  Previous extraction results for chaining (internal use)

  Array items

  • [item]

    string

• limit

  number

  Maximum results to show (default: 10)

Parameters

• job_id

  string

  Required

  Monitoring job ID

Parameters

• enabled_only

  boolean

Parameters

• job_id

  string

  Monitoring job ID (optional - provide URL if no job exists)

• url

  string

  URL to preview (required if job_id not provided)

• disable_js

  boolean

  Disable JavaScript execution (faster but may miss dynamic content)

• preview_size

  string (snippet, partial, full)

  How much data to return: snippet (first 5000 chars), partial (35000 chars), full

  Allowed values: snippet, partial, full

• use_cache

  boolean

  Use cached data if available (faster, default: true). Set to false to force fresh crawl.

Parameters

• job_id

  string

  Required

  Monitoring job ID

• task_id

  string

  Required

  Task ID to test

• limit

  number

  Maximum results to show (default: 5)

• show_steps

  boolean

  Show intermediate results for each step

Parameters

• job_id

  string

  Job ID to test against. Optional when url is provided. Use job to inherit recipe settings (JS, pagination).

• url

  string

  URL to test. Required when job_id is not provided. When both supplied, URL overrides job URL for ad-hoc testing.

• method

  string (xPath, xPath_extract, json, regex, gpt)

  Required

  Extraction method. Use xPath for single nodes, xPath_extract for lists, regex for cleanup, json for transforms/filters, gpt for semantic post-processing.

  Allowed values: xPath, xPath_extract, json, regex, gpt

• pattern

  string

  Required

  Pattern to test. xPath: //div[@class="price"], regex: /\d+/, json: a transform template like ${ title } or full JSON object template, gpt: a clear instruction prompt.

• previous_steps

  array

  Previous steps applied before testing this one. json steps may pass a transform template as pattern or provide jsonConditions.

  Array items

  • [item]

    object

    Nested fields

    • method

      string (xPath, xPath_extract, json, regex, gpt)

      Previous step method

      Allowed values: xPath, xPath_extract, json, regex, gpt

    • pattern

      string

      Previous step pattern (null for json with jsonConditions)

    • jsonConditions

      string

      Optional stringified conditions for json step. Will be decoded/normalized.

• use_cache

  boolean

  Use cached data if available (default: true)

Parameters

• job_id

  string

  Required

  Monitoring job ID to validate

• validation_level

  string (basic, full, production)

  Required

  Validation depth. basic: config only. full: config + connectivity + extraction. production: full + end-to-end + enable on success.

  Allowed values: basic, full, production

• fix_issues

  boolean

  Attempt to automatically fix common issues (default: false)