Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 19 additions & 12 deletions src/content/docs/integrations/ftp.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -4,36 +4,43 @@ title: FTP

Tenzir supports the [File Transfer Protocol
(FTP)](https://en.wikipedia.org/wiki/File_Transfer_Protocol), both downloading
and uploading files.
and uploading files. Use <Op>from_ftp</Op> to download bytes and parse them
with a subpipeline, and use <Op>to_ftp</Op> to print events with a subpipeline
and upload the result.

![FTP](ftp.svg)

FTP consists of two separate TCP connections, one control and one data
connection. This can be tricky for some firewalls and may require special
attention.

:::tip[URL Support]
The URL schemes `ftp://` and `ftps://` dispatch to
<Op>load_ftp</Op> and
<Op>save_ftp</Op> for seamless URL-style use via
<Op>from</Op> and
<Op>to</Op>.
:::

## Examples

### Download a file from an FTP server
These examples use the direct FTP operators with explicit parsing and printing
subpipelines.

### Download and parse a file from an FTP server

Use <Op>from_ftp</Op> with <Op>read_ndjson</Op> to turn the downloaded bytes
into events.

```tql
from "ftp://user:pass@ftp.example.org/path/to/file.json"
from_ftp "ftp://user:pass@ftp.example.org/path/to/file.ndjson" {
read_ndjson
}
```

### Upload events to an FTP server

Use <Op>to_ftp</Op> with <Op>write_ndjson</Op> to serialize events before
uploading them.

```tql
from {
x: 42,
y: "foo",
}
to "ftp://user:pass@ftp.example.org/a/b/c/events.json.gz"
to_ftp "ftp://user:pass@ftp.example.org/a/b/c/events.ndjson" {
write_ndjson
}
```
48 changes: 24 additions & 24 deletions src/content/docs/reference/operators.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -343,6 +343,10 @@ operators:
description: 'Reads one or multiple files from a filesystem.'
example: 'from_file "s3://data/**.json"'
path: 'reference/operators/from_file'
- name: 'from_ftp'
description: 'Downloads bytes via FTP and parses them with a subpipeline.'
example: 'from_ftp "ftp.example.org/events.ndjson" { read_ndjson }'
path: 'reference/operators/from_ftp'
- name: 'from_fluent_bit'
description: 'Receives events via Fluent Bit.'
example: 'from_fluent_bit "opentelemetry"'
Expand Down Expand Up @@ -395,10 +399,6 @@ operators:
description: 'Loads the contents of the file at `path` as a byte stream.'
example: 'load_file "/tmp/data.json"'
path: 'reference/operators/load_file'
- name: 'load_ftp'
description: 'Loads a byte stream via FTP.'
example: 'load_ftp "ftp.example.org"'
path: 'reference/operators/load_ftp'
- name: 'load_gcs'
description: 'Loads bytes from a Google Cloud Storage object.'
example: 'load_gcs "gs://bucket/object.json"'
Expand Down Expand Up @@ -635,10 +635,6 @@ operators:
description: 'Writes a byte stream to a file.'
example: 'save_file "/tmp/out.json"'
path: 'reference/operators/save_file'
- name: 'save_ftp'
description: 'Saves a byte stream via FTP.'
example: 'save_ftp "ftp.example.org"'
path: 'reference/operators/save_ftp'
- name: 'save_gcs'
description: 'Saves bytes to a Google Cloud Storage object.'
example: 'save_gcs "gs://bucket/object.json"'
Expand Down Expand Up @@ -707,6 +703,10 @@ operators:
description: 'Sends events via Fluent Bit.'
example: 'to_fluent_bit "elasticsearch" …'
path: 'reference/operators/to_fluent_bit'
- name: 'to_ftp'
description: 'Prints events to bytes and uploads them via FTP.'
example: 'to_ftp "ftp.example.org/events.ndjson" { write_ndjson }'
path: 'reference/operators/to_ftp'
- name: 'to_google_cloud_logging'
description: 'Sends events to Google Cloud Logging.'
example: 'to_google_cloud_logging …'
Expand Down Expand Up @@ -2018,14 +2018,6 @@ load_file "/tmp/data.json"

</ReferenceCard>

<ReferenceCard title="load_ftp" description="Loads a byte stream via FTP." href="/reference/operators/load_ftp">

```tql
load_ftp "ftp.example.org"
```

</ReferenceCard>

<ReferenceCard title="load_gcs" description="Loads bytes from a Google Cloud Storage object." href="/reference/operators/load_gcs">

```tql
Expand Down Expand Up @@ -2143,6 +2135,14 @@ from_file "s3://data/**.json"

</ReferenceCard>

<ReferenceCard title="from_ftp" description="Downloads bytes via FTP and parses them with a subpipeline." href="/reference/operators/from_ftp">

```tql
from_ftp "ftp.example.org/events.ndjson" { read_ndjson }
```

</ReferenceCard>

<ReferenceCard title="from_fluent_bit" description="Receives events via Fluent Bit." href="/reference/operators/from_fluent_bit">

```tql
Expand Down Expand Up @@ -2356,14 +2356,6 @@ save_file "/tmp/out.json"

</ReferenceCard>

<ReferenceCard title="save_ftp" description="Saves a byte stream via FTP." href="/reference/operators/save_ftp">

```tql
save_ftp "ftp.example.org"
```

</ReferenceCard>

<ReferenceCard title="save_gcs" description="Saves bytes to a Google Cloud Storage object." href="/reference/operators/save_gcs">

```tql
Expand Down Expand Up @@ -2489,6 +2481,14 @@ to_fluent_bit "elasticsearch" …

</ReferenceCard>

<ReferenceCard title="to_ftp" description="Prints events to bytes and uploads them via FTP." href="/reference/operators/to_ftp">

```tql
to_ftp "ftp.example.org/events.ndjson" { write_ndjson }
```

</ReferenceCard>

<ReferenceCard title="to_google_cloud_logging" description="Sends events to Google Cloud Logging." href="/reference/operators/to_google_cloud_logging">

```tql
Expand Down
1 change: 0 additions & 1 deletion src/content/docs/reference/operators/from.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,6 @@ load_tcp "tcp://0.0.0.0:12345", parallel=10 {
| `elasticsearch` | <Op>from_opensearch</Op> | `from "elasticsearch://1.2.3.4:9200` |
| `file` | <Op>load_file</Op> | `from "file://path/to/file.json"` |
| `fluent-bit` | <Op>from_fluent_bit</Op> | `from "fluent-bit://elasticsearch"` |
| `ftp`, `ftps` | <Op>load_ftp</Op> | `from "ftp://example.com/file.json"` |
| `gs` | <Op>load_gcs</Op> | `from "gs://bucket/object.json"` |
| `http`, `https` | <Op>load_http</Op> | `from "http://example.com/file.json"` |
| `inproc` | <Op>load_zmq</Op> | `from "inproc://127.0.0.1:56789" { read_json }` |
Expand Down
68 changes: 68 additions & 0 deletions src/content/docs/reference/operators/from_ftp.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
---
title: from_ftp
category: Inputs/Events
example: 'from_ftp "ftp.example.org/events.ndjson" { read_ndjson }'
---

Downloads bytes via FTP or FTPS and parses them with a subpipeline.

```tql
from_ftp url:string, [tls=record] { … }
```

## Description

The `from_ftp` operator downloads bytes from an FTP or FTPS server and forwards
them to the required subpipeline.

### `url: string`

The URL to request from. You can omit the `ftp://` scheme.

### `tls = record (optional)`

TLS configuration.

By default, `ftps://` enables TLS and `ftp://` does not. If you omit the
scheme, the operator assumes `ftp://`.

import TLSOptions from '@partials/operators/TLSOptions.mdx';

<TLSOptions />

### `{ … }`

A required parsing subpipeline.

The subpipeline receives the downloaded body as bytes and must return events.
For example, use <Op>read_ndjson</Op> to parse newline-delimited JSON.

## Examples

### Download NDJSON from an FTP server

Use <Op>read_ndjson</Op> when the remote file already contains
newline-delimited JSON.

```tql
from_ftp "ftp://user:pass@ftp.example.org/events.ndjson" {
read_ndjson
}
```

### Download gzipped JSON and decompress it explicitly

Decompress the downloaded bytes before parsing them when the remote file is
stored as gzip-compressed JSON.

```tql
from_ftp "ftp://user:pass@ftp.example.org/events.json.gz" {
decompress gzip
read_json
}
```

## See Also

- <Op>to_ftp</Op>
- <Integration>ftp</Integration>
34 changes: 0 additions & 34 deletions src/content/docs/reference/operators/load_ftp.mdx

This file was deleted.

34 changes: 0 additions & 34 deletions src/content/docs/reference/operators/save_ftp.mdx

This file was deleted.

1 change: 0 additions & 1 deletion src/content/docs/reference/operators/to.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,6 @@ If no scheme is present, the connector attempts to save to the local filesystem.
| `elasticsearch` | <Op>to_opensearch</Op> | `to "elasticsearch://…` |
| `file` | <Op>save_file</Op> | `to "file://path/to/file.json"` |
| `fluent-bit` | <Op>to_fluent_bit</Op> | `to "fluent-bit://elasticsearch"` |
| `ftp`, `ftps` | <Op>save_ftp</Op> | `to "ftp://example.com/file.json"` |
| `gs` | <Op>save_gcs</Op> | `to "gs://bucket/object.json"` |
| `http`, `https` | <Op>save_http</Op> | `to "http://example.com/file.json"` |
| `inproc` | <Op>save_zmq</Op> | `to "inproc://127.0.0.1:56789" { write_json }` |
Expand Down
77 changes: 77 additions & 0 deletions src/content/docs/reference/operators/to_ftp.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
---
title: to_ftp
category: Outputs/Events
example: 'to_ftp "ftp.example.org/events.ndjson" { write_ndjson }'
---

Prints events to bytes and uploads them via FTP or FTPS.

```tql
to_ftp url:string, [tls=record] { … }
```

## Description

The `to_ftp` operator sends events to an FTP or FTPS server.

The required subpipeline receives events and must return bytes.

### `url: string`

The URL to upload to. You can omit the `ftp://` scheme.

### `tls = record (optional)`

TLS configuration.

By default, `ftps://` enables TLS and `ftp://` does not. If you omit the
scheme, the operator assumes `ftp://`.

import TLSOptions from '@partials/operators/TLSOptions.mdx';

<TLSOptions />

### `{ … }`

A required printing subpipeline.

The subpipeline receives events and must return bytes. For example, use
<Op>write_ndjson</Op> to serialize events as newline-delimited JSON.

## Examples

### Upload events as NDJSON

Use <Op>write_ndjson</Op> to serialize each event as one JSON object per line
before uploading it.

```tql
from {
x: 42,
y: "foo",
}
to_ftp "ftp://user:pass@ftp.example.org/events.ndjson" {
write_ndjson
}
```

### Upload compressed NDJSON

Add <Op>compress gzip</Op> to the printing subpipeline when you want to upload
compressed output.

```tql
from {
x: 42,
y: "foo",
}
to_ftp "ftp://user:pass@ftp.example.org/events.ndjson.gz" {
write_ndjson
compress gzip
}
```

## See Also

- <Op>from_ftp</Op>
- <Integration>ftp</Integration>
Loading