Skip to content

Enable filters.python to specify which cloud dimensions will be returned in the pipeline arrays #175

Closed

Description

@robeckgeo

Recommendation:

  • Provide an alternative method/function to access the arrays. The new method would include an optional iterable argument for the requested dimensions, defaulting to all dimensions.
  • Or provide an optional list element in the Python filter configuration to identify which dimensions are to be returned in the array. The new option would be valid whether the filter’s “function” option is applied or not. The filter would then be used to a) provide a callback function and/or b) identify which arrays to return.
  • Or create another Python filter solely for the purpose of identifying which dimensions to return in the arrays.

Justification:

  • Any solution to reduce memory usage could significantly improve the chances that a containerized process, like Docker, would not fail due to exceeding allocated memory.
  • A streamable pipeline could be used to help reduce the memory used for the returned arrays. However, when using a Python filter, among other filters, the PDAL pipeline is no longer streamable. This results in the entire dataset returned. That dataset can be extremely large when it contains hundreds of millions of points and has dozens of dimensions.

Activity

  1. abellgithub commented on Aug 21, 2024

    @abellgithub
    Collaborator

    I don't understand. Only the dimensions that are referenced are copied.

  2. robeckgeo commented on Aug 21, 2024

    @robeckgeo
    Author

    The requested enhancement is for the returned pipeline arrays, i.e., Pipeline.arrays. To best of my knowledge, you cannot dictate dimensions while accessing the arrays property/attribute; only afterwards, i.e., arrays[0][[list of dimensions]].

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      ⌂ Proxy