Self-hosted alternatives to bigquery-pipeline-audit

B

bigquery-pipeline-audit

Audits Python + BigQuery pipelines for cost safety, idempotency

bigquery-pipeline-audit is an agent skill from the github/awesome-copilot collection that audits a Python and BigQuery data pipeline for cost safety, idempotency and production readiness. The agent acts as a senior data engineer reviewing the script. It locates every BigQuery job trigger, such as client.query, load_table_from_*, extract_table and copy_table, along with DDL and DML statements, and every external call to APIs, LLM services or storage, to show what will actually be billed. It then checks for loop-driven query multiplication, where a query runs once per item in a loop and the cost grows with the data, and for missing maximum_bytes_billed limits that would let a query scan more data than intended. The audit also checks whether a rerun would corrupt data, and whether failures are visible rather than silently swallowed. The response follows a fixed structure, sections A to F and a final summary, and it references exact function names and line locations with minimal suggested fixes rather than rewrites. This makes the audit practical to act on in a code review. The findings should be verified against the pipeline's real workload, since the cost exposure depends on data volumes that the code alone does not show. The skill is installed with the skills CLI from the github/awesome-copilot repository. Its skills.sh page showed about 8.8K installs and passing results from three security audits at the time of the scan.