Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions src/content/docs/guides/testing/write-tests.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -354,7 +354,7 @@ key and opt into graceful skipping with `skip: {on: capability-unavailable}`:
# tests/gcs/test.yaml
suite: gcs-smoke
requires:
operators: [from_gcs]
operators: [from_google_cloud_storage]
skip:
on: capability-unavailable
```
Expand Down Expand Up @@ -382,7 +382,7 @@ You now have a project that owns its inputs, tests, fixtures, and baselines.
From here you can:

- <Guide>testing/run-tests</Guide> to learn about executing the suite,
selecting tests, and setting up CI.
selecting tests, and setting up CI.
- Add custom runners under `runners/` when you need specialized logic around
`tenzir` invocations.
- Build Python fixtures that publish or verify data through the helper APIs in
Expand Down
6 changes: 3 additions & 3 deletions src/content/docs/integrations/google/cloud-storage.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -14,20 +14,20 @@ Default Credentials](https://google.aip.dev/auth/4110).

## Examples

Use <Op>from_gcs</Op> to read files from Cloud
Use <Op>from_google_cloud_storage</Op> to read files from Cloud
Storage. It supports glob patterns and automatic format detection. For writing,
use <Op>save_gcs</Op> with a print operator.

### Read events from a file in a bucket

```tql
from_gcs "gs://bucket/path/to/file.json"
from_google_cloud_storage "gs://bucket/path/to/file.json"
```

### Read all JSON files from a bucket

```tql
from_gcs "gs://bucket/logs/**.json"
from_google_cloud_storage "gs://bucket/logs/**.json"
```

### Write an event to a file in a bucket
Expand Down
64 changes: 56 additions & 8 deletions src/content/docs/reference/operators.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -363,14 +363,14 @@ operators:
description: 'Receives events via Fluent Bit.'
example: 'from_fluent_bit "opentelemetry"'
path: 'reference/operators/from_fluent_bit'
- name: 'from_gcs'
description: 'Reads one or multiple files from Google Cloud Storage.'
example: 'from_gcs "gs://my-bucket/data/**.json"'
path: 'reference/operators/from_gcs'
- name: 'from_google_cloud_pubsub'
description: 'Subscribes to a Google Cloud Pub/Sub subscription and yields events.'
example: 'from_google_cloud_pubsub project_id="my-project", subscription_id="my-sub"'
path: 'reference/operators/from_google_cloud_pubsub'
- name: 'from_google_cloud_storage'
description: 'Reads one or multiple files from Google Cloud Storage.'
example: 'from_google_cloud_storage "gs://my-bucket/data/**.json"'
path: 'reference/operators/from_google_cloud_storage'
- name: 'from_http'
description: 'Sends and receives HTTP/1.1 requests.'
example: 'from_http "0.0.0.0:8080"'
Expand Down Expand Up @@ -735,6 +735,10 @@ operators:
description: 'Sends OCSF events to Amazon Security Lake.'
example: 'to_amazon_security_lake "s3://…"'
path: 'reference/operators/to_amazon_security_lake'
- name: 'to_azure_blob_storage'
description: 'Writes events to one or multiple blobs in Azure Blob Storage.'
example: 'to_azure_blob_storage "abfs://container/data/{uuid}.json" { write_ndjson }'
path: 'reference/operators/to_azure_blob_storage'
- name: 'to_azure_log_analytics'
description: 'Sends events to the Microsoft Azure Logs Ingestion API.'
example: 'to_azure_log_analytics tenant_id="...", workspace_id="..."'
Expand All @@ -743,6 +747,10 @@ operators:
description: 'Sends events to a ClickHouse table.'
example: 'to_clickhouse table="my_table"'
path: 'reference/operators/to_clickhouse'
- name: 'to_file'
description: 'Writes events to one or multiple files on a filesystem.'
example: 'to_file "/tmp/out.json" { write_ndjson }'
path: 'reference/operators/to_file'
- name: 'to_fluent_bit'
description: 'Sends events via Fluent Bit.'
example: 'to_fluent_bit "elasticsearch" …'
Expand All @@ -755,6 +763,10 @@ operators:
description: 'Publishes events to a Google Cloud Pub/Sub topic.'
example: 'to_google_cloud_pubsub project_id="my-project", topic_id="alerts", message=text'
path: 'reference/operators/to_google_cloud_pubsub'
- name: 'to_google_cloud_storage'
description: 'Writes events to one or multiple objects in Google Cloud Storage.'
example: 'to_google_cloud_storage "gs://my-bucket/data/{uuid}.json" { write_ndjson }'
path: 'reference/operators/to_google_cloud_storage'
- name: 'to_google_secops'
description: 'Sends unstructured events to a Google SecOps Chronicle instance.'
example: 'to_google_secops …'
Expand All @@ -775,6 +787,10 @@ operators:
description: 'Sends events to an OpenSearch-compatible Bulk API.'
example: 'to_opensearch "localhost:9200", …'
path: 'reference/operators/to_opensearch'
- name: 'to_s3'
description: 'Writes events to one or multiple objects in Amazon S3.'
example: 'to_s3 "s3://my-bucket/data/{uuid}.json" { write_ndjson }'
path: 'reference/operators/to_s3'
- name: 'to_sentinelone_data_lake'
description: 'Sends security events to SentinelOne Singularity Data Lake via REST API.'
example: 'to_sentinelone_data_lake "https://…", …'
Expand Down Expand Up @@ -2239,18 +2255,18 @@ from_fluent_bit "opentelemetry"

</ReferenceCard>

<ReferenceCard title="from_gcs" description="Reads one or multiple files from Google Cloud Storage." href="/reference/operators/from_gcs">
<ReferenceCard title="from_google_cloud_pubsub" description="Subscribes to a Google Cloud Pub/Sub subscription and yields events." href="/reference/operators/from_google_cloud_pubsub">

```tql
from_gcs "gs://my-bucket/data/**.json"
from_google_cloud_pubsub project_id="my-project", subscription_id="my-sub"
```

</ReferenceCard>

<ReferenceCard title="from_google_cloud_pubsub" description="Subscribes to a Google Cloud Pub/Sub subscription and yields events." href="/reference/operators/from_google_cloud_pubsub">
<ReferenceCard title="from_google_cloud_storage" description="Reads one or multiple files from Google Cloud Storage." href="/reference/operators/from_google_cloud_storage">

```tql
from_google_cloud_pubsub project_id="my-project", subscription_id="my-sub"
from_google_cloud_storage "gs://my-bucket/data/**.json"
```

</ReferenceCard>
Expand Down Expand Up @@ -2595,6 +2611,14 @@ to_amazon_security_lake "s3://…"

</ReferenceCard>

<ReferenceCard title="to_azure_blob_storage" description="Writes events to one or multiple blobs in Azure Blob Storage." href="/reference/operators/to_azure_blob_storage">

```tql
to_azure_blob_storage "abfs://container/data/{uuid}.json" { write_ndjson }
```

</ReferenceCard>

<ReferenceCard title="to_azure_log_analytics" description="Sends events to the Microsoft Azure Logs Ingestion API." href="/reference/operators/to_azure_log_analytics">

```tql
Expand All @@ -2611,6 +2635,14 @@ to_clickhouse table="my_table"

</ReferenceCard>

<ReferenceCard title="to_file" description="Writes events to one or multiple files on a filesystem." href="/reference/operators/to_file">

```tql
to_file "/tmp/out.json" { write_ndjson }
```

</ReferenceCard>

<ReferenceCard title="to_fluent_bit" description="Sends events via Fluent Bit." href="/reference/operators/to_fluent_bit">

```tql
Expand All @@ -2635,6 +2667,14 @@ to_google_cloud_pubsub project_id="my-project", topic_id="alerts", message=text

</ReferenceCard>

<ReferenceCard title="to_google_cloud_storage" description="Writes events to one or multiple objects in Google Cloud Storage." href="/reference/operators/to_google_cloud_storage">

```tql
to_google_cloud_storage "gs://my-bucket/data/{uuid}.json" { write_ndjson }
```

</ReferenceCard>

<ReferenceCard title="to_google_secops" description="Sends unstructured events to a Google SecOps Chronicle instance." href="/reference/operators/to_google_secops">

```tql
Expand Down Expand Up @@ -2675,6 +2715,14 @@ to_opensearch "localhost:9200", …

</ReferenceCard>

<ReferenceCard title="to_s3" description="Writes events to one or multiple objects in Amazon S3." href="/reference/operators/to_s3">

```tql
to_s3 "s3://my-bucket/data/{uuid}.json" { write_ndjson }
```

</ReferenceCard>

<ReferenceCard title="to_sentinelone_data_lake" description="Sends security events to SentinelOne Singularity Data Lake via REST API." href="/reference/operators/to_sentinelone_data_lake">

```tql
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Reads one or multiple files from Azure Blob Storage.

```tql
from_azure_blob_storage url:string, [account_key=string, watch=bool,
remove=bool, rename=string->string, path_field=field, max_age=duration] { … }
remove=bool, rename=string->string, max_age=duration] { … }
```

## Description
Expand Down Expand Up @@ -90,7 +90,10 @@ from_azure_blob_storage "abfs://input/**.json",
### Add source path to events

```tql
from_azure_blob_storage "abfs://data/**.json", path_field=source_file
from_azure_blob_storage "abfs://data/**.json" {
read_json
source_file = $file.path
}
```

## See Also
Expand Down
2 changes: 1 addition & 1 deletion src/content/docs/reference/operators/from_file.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Reads one or multiple files from a filesystem.

```tql
from_file url:string, [watch=bool, remove=bool, rename=string->string,
path_field=field, max_age=duration, mmap=bool] { … }
max_age=duration, mmap=bool] { … }
```

## Description
Expand Down
Original file line number Diff line number Diff line change
@@ -1,21 +1,21 @@
---
title: from_gcs
title: from_google_cloud_storage
category: Inputs/Events
example: 'from_gcs "gs://my-bucket/data/**.json"'
example: 'from_google_cloud_storage "gs://my-bucket/data/**.json"'
---

import FromFileCommonParams from '@partials/operators/FromFileCommonParams.mdx';

Reads one or multiple files from Google Cloud Storage.

```tql
from_gcs url:string, [anonymous=bool, watch=bool, remove=bool,
rename=string->string, path_field=field, max_age=duration] { … }
from_google_cloud_storage url:string, [anonymous=bool, watch=bool, remove=bool,
rename=string->string, max_age=duration] { … }
```

## Description

The `from_gcs` operator reads files from Google Cloud Storage, with support for
The `from_google_cloud_storage` operator reads files from Google Cloud Storage, with support for
glob patterns, automatic format detection, and file monitoring.

By default, authentication is handled by Google's Application Default
Expand Down Expand Up @@ -56,33 +56,36 @@ Defaults to `false`.
### Read every JSON file from a bucket

```tql
from_gcs "gs://my-bucket/data/**.json"
from_google_cloud_storage "gs://my-bucket/data/**.json"
```

### Read CSV files from a public bucket

```tql
from_gcs "gs://public-dataset/data.csv", anonymous=true
from_google_cloud_storage "gs://public-dataset/data.csv", anonymous=true
```

### Read Zeek logs continuously

```tql
from_gcs "gs://logs/zeek/**.log", watch=true {
from_google_cloud_storage "gs://logs/zeek/**.log", watch=true {
read_zeek_tsv
}
```

### Add source path to events

```tql
from_gcs "gs://data-bucket/**.json", path_field=source_file
from_google_cloud_storage "gs://data-bucket/**.json" {
read_json
source_file = $file.path
}
```

### Read Suricata EVE JSON logs with custom parsing

```tql
from_gcs "gs://security-logs/suricata/**.json" {
from_google_cloud_storage "gs://security-logs/suricata/**.json" {
read_suricata
}
```
Expand Down
7 changes: 5 additions & 2 deletions src/content/docs/reference/operators/from_s3.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Reads one or multiple files from Amazon S3.

```tql
from_s3 url:string, [anonymous=bool, aws_iam=record, watch=bool,
remove=bool, rename=string->string, path_field=field, max_age=duration] { … }
remove=bool, rename=string->string, max_age=duration] { … }
```

## Description
Expand Down Expand Up @@ -98,7 +98,10 @@ from_s3 "s3://input-bucket/**.json",
### Add source path to events

```tql
from_s3 "s3://data-bucket/**.json", path_field=source_file
from_s3 "s3://data-bucket/**.json" {
read_json
source_file = $file.path
}
```

### Read Zeek logs with anonymous access
Expand Down
Loading
Loading