Repository navigation
Conversation
Repositories upload an SBOM of their default branch, for example from
syft in CI, to PUT /api/projects/{project}/sbom. The proxy records the
package versions it lists, keyed by the same unversioned PURL the
handlers record packages under, so the inventory joins with what the
proxy has cached and the vulnerabilities it holds.
That answers which repository uses which version of a package: the new
/ui/projects pages list each project's package versions with outdated
and vulnerability markers, package pages show the projects per version,
the dashboard lists vulnerable versions in use, and
GET /api/projects?purl= does the same for scripts.
The feature is off by default. Uploads and deletes need projects.token;
an upload replaces the project's components in one transaction.
projects.max_age removes projects that stopped uploading.
A syft SBOM of a repository lists the repository itself: the module in go.mod and the root of package-lock.json, both without a version or with the version from package.json, and the module its Go binaries were built from. The project page showed them as dependencies. Entries without a version are now skipped, as are lock-file roots (lock entries that depend on others while nothing depends on them) and modules syft marks as a binary's main module.
The page asks for the upload token and sends it to DELETE
/api/projects/{project}, so /ui gains no unauthenticated way to delete.
The token is kept in session storage for the next delete.
The vulnerabilities table is only written by tests: nothing in the proxy calls UpsertVulnerability, so the dashboard card, the per-project count and the per-component markers were always empty. Remove them along with GetComponentVulnerabilities and inventory.Affected, and keep the outdated marker, which uses the latest version the handlers record.
FromPURL now names packages the way their handlers store them: - NuGet IDs are lowercased, as the NuGet handler stores them. - PyPI names are normalized as in PEP 503. The PyPI handler takes the name from the file name, which spells "." as "_", so pkg:pypi/zope.interface has to match the cached zope-interface. - Conda package URLs without a channel qualifier are skipped: the conda handler records packages as channel/name, so they can never match. - The Go main-module check ignores case. syft keeps the module path's case in its mainModule property, while golang package URLs are lowercased. A table test checks every mapped PURL type against the name its handler passes when caching.
- GetProjectsUsingPackage no longer counts each project's components
per row; the package page only needs names and versions. The ?purl=
listing leaves component_count out accordingly.
- The delete button sends to DELETE /ui/api/projects/{project}, the same
handler as the API and still behind the token, so it sits under /ui
like the cache deletes.
- The project page answers 500 when its components can't be read,
instead of showing an empty project.
- DELETE of a path ending in /sbom that names no project answers 400
with a hint, since it's most likely the upload path.
- The upload endpoint lists JSON, XML and plain text (SPDX tag-value)
as accepted.
- The docs say that versions are compared as the SBOM spells them.
andrew
left a comment
There was a problem hiding this comment.
Before getting into the details, one point about how this fits with the other git-pkgs modules. When the proxy needs something a git-pkgs library gets wrong or doesn't provide, I want it fixed in that library, tagged, and bumped here, not carried as a local version in the proxy. Otherwise we end up with two implementations that drift apart, and every other consumer of the library keeps the bug.
This applies in three places in this PR.
lockFileRoots in internal/inventory/inventory.go does the same root inference as sbom.ClassifyScope, and both get it wrong for lock files that don't list the project. With a syft SBOM of a poetry.lock, requests is a direct dependency that depends on urllib3, and it gets treated as the project's own entry and skipped. I've opened git-pkgs/sbom#23 with a repro and a proposed fix: keep the metadata.component bom-ref so a declared root can be used. Root detection should be fixed there, and the inventory should use what sbom provides instead of its own heuristic.
The case-insensitive main-module comparison works around git-pkgs/purl#53, where purl v0.1.24 lowercases Go module paths. It's fixed in git-pkgs/purl#54 but not tagged yet. Once there's a release, bump purl here and drop the strings.ToLower along with the comment about golang PURLs being lowercased. The inventory stores those lowercased PURLs in project_components, so this should land before the inventory ships, not after.
normalizePyPIName repeats the PEP 503 folding in purl.ComparisonForm(), and the NuGet lowercasing in handlerName is the same as what ComparisonForm does. Use purl for those. If mapping a PURL to the name a handler stores needs something purl doesn't have yet, add it to purl.
The same goes for code already in the proxy. internal/mirror/source.go already maps SBOM PURLs to the ecosystem and name the handlers use (handlerEcosystem, condaName), and FromPURL is a second mapping that disagrees with it: the mirror keeps the distro namespace for apk/deb/rpm (alpine/busybox) and doesn't normalize PyPI or NuGet names, while FromPURL drops the namespace and does both. There should be one helper, used by both the mirror and the inventory, with the disagreement settled and the table test covering it.
# Conflicts: # internal/server/dashboard.go
packageurl.HandlerPackage returns the ecosystem and name the handlers store a package URL under. The mirror and the inventory both use it, which settles where they disagreed: distribution namespaces of apk, alpm, bsd, deb and rpm are dropped, PyPI and NuGet names use purl's comparison form, and Maven names are group:artifact as the handler stores them, where the mirror used group/artifact.
|
Merged main and moved the PURL mapping into one helper, Bumping purl to v0.2.1 on its own breaks the Swift registry keys and Go retention matching that #642 handles, so I'll merge main again once #642 is in, then bump and drop the |
With purl v0.2.1, FullName returns a Swift package as its registry identity, scope.name. The mirror splits namespace/name again to find the scope or the source repository, so HandlerPackage keeps that form, as mirrorTarget did before the two mappings were shared.
purl v0.2.1 keeps the case of Go module paths, so the inventory no longer lowercases syft's mainModule. Module paths are case-sensitive: github.com/azure/foo is a dependency of github.com/Azure/foo, not part of it.
|
Merged main after #642, so this is on purl v0.2.1 now. Two follow-ups:
|
FuzzParse checks that an uploaded body never makes the inventory panic and that every component it records has a package URL, ecosystem and version. TestSaveProjectConcurrentUploads saves one project from eight uploads at once and checks that each succeeds and the components of exactly one remain.
This adds an opt-in project inventory that answers "which repository uses which version of a package?". Repositories upload an SBOM of their default branch (for example
syft dir:. -o cyclonedx-jsonin a CI job) toPUT /api/projects/{project}/sbom, and the proxy records the package versions it lists./ui/projectslists the projects and, per project, its package versions with an outdated marker when the proxy knows a newer version. Package pages get a "Used by" section listing the projects per version, including versions the proxy hasn't cached.GET /api/projects?purl=pkg:npm/lodash@4.17.20answers the same question for scripts.Why: the proxy knows which packages are cached, but not who depends on them. When an advisory comes out, the question is which repositories to fix. Downloads through the proxy can't answer that (CI caches, lock files resolved elsewhere, other registries), but an SBOM from each repository can.
How:
projects.enabled,projects.token(required when enabled,${VAR}andPROXY_PROJECTS_TOKENsupported) andprojects.max_age(e.g.90d, parsed like retention durations; empty keeps projects). Uploads and deletes use the existingrequireBearerToken; reads are open like the rest of/api. Disabled, none of the routes or pages exist.internal/inventoryparses the SBOM withgit-pkgs/sbom(CycloneDX JSON/XML, SPDX JSON/tag-value) and maps each package URL to the ecosystem name and unversioned PURL the handlers record packages under (packageurl.MakeCacheStrings), so components joinpackages.purl. Names are spelled the way each handler stores them: distro types (apk,deb,rpm,alpm,bsd) drop the distro namespace, NuGet IDs are lowercased, PyPI names are normalized as in PEP 503, conda becomeschannel/name. A table test checks every mapped type against the name its handler passes when caching. Entries without a PURL, of a type the proxy doesn't serve, without a version, and conda entries without achannelqualifier are counted as skipped; duplicates are recorded once. The project's own package is skipped too: the root of a lock file such aspackage-lock.json(a lock entry that depends on others while nothing depends on it) and, with syft, the Go module its binaries were built from (syft:metadata:mainModule, compared case-insensitively). Without this, a repository listed itself as its own dependency.020_add_projectsaddsprojects(unique name, ref, branch, uploaded_at) andproject_components(project_id, package_purl, ecosystem, name, version), indexed by(package_purl, version). An upload replaces the project's components in one transaction; the project row is upserted first, so concurrent uploads and pruning of the same project take turns.[A-Za-z0-9._-], no empty,.or..segments, at most 255 characters. They are accepted as path segments or URL-encoded as one segment (group%2Fsub%2Frepo); the wildcard is decoded only when chi routed on the raw path, like the package routes. Bodies are capped at 32 MiB. A DELETE of a path ending in/sbomthat names no project answers 400 and points toDELETE /api/projects/{project}.git-pkgs/versusing the ecosystem's scheme. Versions are taken as the SBOM spells them, so a different spelling (Composerv6.4.1vs6.4.1, Debian epochs) shows as "not cached" on the package page; the docs say so.runExclusive.DELETE /ui/api/projects/{project}, the same handler as the API behind the same token, under/uilike the cache deletes so ingress rules for the UI cover it. The token is kept in session storage.How tested: unit tests for SBOM mapping (a real syft CycloneDX output with npm, Composer, Go and Alpine packages, plus files and the OS without PURLs; the per-type handler name table; PyPI normalization; conda without a channel; a mixed-case Go main module), name validation and version comparison; database tests for replace semantics, lookup by package with and without version, pruning and the migration, on SQLite and Postgres; API tests for auth (missing and wrong token) on both delete routes, encoded and raw project paths, path traversal and invalid names, DELETE of the upload path, body limit and bad bodies, purl lookups; UI tests for the project page, its error case and "Used by"; and a test that the routes follow
projects.enabledon the real server. Also tried by hand: uploaded the syft output, cachedlodashthrough/npm, and checked the project and package pages. I've also run it on my own instance with syft 1.55 SBOMs of four repositories (Go, npm and PyPI) uploaded the way the CI job in the docs does.Possible follow-up: per-project vulnerabilities. The
vulnerabilitiestable isn't filled outside tests, so this PR doesn't show any. Matching a project's versions against OSV's affected ranges would work without it.Open questions:
/uiand/api/projects. The docs say so and suggest restricting both at the ingress; should reads require the token instead?020_add_projects; Serve dated snapshots of Alpine APK indexes #622 also adds a 020, so whichever merges second needs renumbering.