Policy engine workflow for building and testing 2nd party plugins

[johnandersen777]

• TODO
  • beta attestation API
  • test cases
    • https://github.com/search?q=org%3Aintel+path%3Apyproject.toml&type=code
      • These are existing intel repos which are pep-518 comparable, so they should build with the build package
  • gen from logs for applicable python versions
• Future
  • overlay support
  • Refactor github_api.py to help with 2nd party build and test #1615
  • Make a single python script run
    • Leverage pin_command() with cached dag workflow context off SCITT
  • Logs to assistants API
    • What failures are we seeing for 2nd party?

[johnandersen777]

Test Cases

• https://github.com/search?q=org%3Aintel+path%3Apyproject.toml&type=code

gh search code --filename pyproject.toml --owner $USER --json repository,path,sha,url

[                                              
  {                   
    "path": "pyproject.toml",
    "repository": {                           
      "id": "MDEwOlJlcG9zaXRvcnk5ODQ2ODEzMw==",       
      "isFork": false,
      "isPrivate": false,                             
      "nameWithOwner": "pdxjohnny/httptest",                                                                                                                   
      "url": "https://github.com/pdxjohnny/httptest"
    },
    "sha": "17b123594123d648e8a064fb0761a248ed0b312e",
    "url": "https://github.com/pdxjohnny/httptest/blob/2239666733f9e77221b01425c8cc71fb18062eb4/pyproject.toml"
  }
]

$ for i in $(gh search code --filename pyproject.toml --owner $USER --json repository,path,sha,url --jq '.[]'); do export REPO_NAME_WITH_OWNER=$(echo "${i}" | jq -r .repository.nameWithOwner); echo $REPO_NAME_WITH_OWNER; done
pdxjohnny/httptest
pdxjohnny/signpost
pdxjohnny/autoentrypoint
pdxjohnny/dffml-operations-flights
pdxjohnny/dffml-operations-github
pdxjohnny/dotfiles
pdxjohnny/noteserver
pdxjohnny/consoletest
pdxjohnny/dffml-operations-us-reps
pdxjohnny/test-ffmpeg
pdxjohnny/dffml-operations-dockerhub

• ci: testing: Enable 2nd party downstream revalidation #1616

for i in $(gh search code --filename pyproject.toml --owner $USER --json repository,path,sha,url --jq '.[]'); do export REPO_NAME_WITH_OWNER=$(echo "${i}" | jq -r .repository.nameWithOwner); gh workflow run --ref 2nd-party-shared-ci-flows -F name-with-owner=$REPO_NAME_WITH_OWNER testing.yml; break; done

• successful workflow run creating attestations for httptest
  • https://github.com/intel/dffml/actions/runs/9690942681

• Dispatched set at 22:05 PDT
  • 4 failures - 3 due to private repos
    • https://github.com/intel/dffml/actions/runs/9690978761/job/26741728126
      • ERROR: file:///home/runner/work/dffml/dffml does not appear to be a Python project: neither 'setup.py' nor 'pyproject.toml' found.
• On intel/
  • 5 success with python 3.12
    • https://github.com/intel/trustauthority-client-for-python

      • Intel Trust Authority Client provides a set of Python modules for attesting different TEEs with Intel Trust Authority. Users can import the Python packages into their application and make REST calls to Intel Trust Authority for fetching token containing information about the TEE attested that can be verified.

• https://chatgpt.com/share/0fda008d-d259-47b0-9434-239618ae7f66

gh search code --filename pyproject.toml --owner $USER --json repository,path,sha,url | python -um json.tool | tee $USER-pep-518-repos.json && export GH_ACCESS_TOKEN=$(gh auth token) && (set -x; for i in $(cat $USER-pep-518-repos.json | jq -c '.[]'); do   export REPO_NAME_WITH_OWNER=$(echo "${i}" | jq -r .repository.nameWithOwner) && export GH_TOKEN=$(gh auth token) && export START_DATE="2024-06-27T00:00:00Z" && export END_DATE="2024-06-27T00:00:00Z" && python -u $(find . -name github_api.py); break;  done; )