# Upload Data to S3 Using Funnel and Amazon Data Firehose

Store device data sent through Amazon Data Firehose in Amazon S3.

## Introduction

In this guide, we create an Amazon Data Firehose instance and an S3 bucket on AWS CloudFormation, then use the Soracom Funnel Amazon Data Firehose adapter to send data directly from your device to an S3 bucket.

We will be using **[Soracom Funnel](https://docs.soracom.io/en/services/funnel)** along with the following AWS resources:

- **[Amazon Data Firehose](https://aws.amazon.com/kinesis/data-firehose/)**
- **[Amazon S3](https://aws.amazon.com/s3/)**
- **[AWS IAM](https://aws.amazon.com/iam/)**
- **[Amazon CloudWatch Logs](https://aws.amazon.com/cloudwatch/)**
- **[AWS CloudFormation](https://aws.amazon.com/cloudformation/)**

Click the links to read about these services in more depth.

### Data Flow

1. The Data sent from your devices will be received by Soracom Funnel.
2. Funnel uses IAM Role authentication to send your data over to Amazon Data Firehose.
3. Amazon Data Firehose relays the data to S3.
4. When the data transfer to S3 fails, the error log is recorded on Amazon CloudWatch Logs, which will be available for 14 days after the error occurs.

![Funnel](https://docs.soracom.io/_astro/step0-1.LzxI3ZO3_Z2mSmLN.webp)

> [!NOTE]
>
> Funnel uses TLS 1.2. Ensure that you configure your cloud service to support TLS 1.2.

## Step 1: Set up AWS Resources

You will first use AWS CloudFormation to configure and prepare all the AWS Resources used on this guide.

### Prepare AWS Resources Using CloudFormation

1. **[Click here](https://console.aws.amazon.com/cloudformation/home?region=ap-northeast-1#/stacks/create/review?stackName=SoracomAwsCfnSampleFirehoseToS3\&templateURL=https:%2F%2Fsoracom-files.s3.amazonaws.com%2Fcfn-samples%2FSoracomAwsCfnSampleFirehoseToS3Stack.yml)** to Use the AWS CloudFormation Template.

> [!NOTE]
>
> This template will default to the Tokyo Region. Change your region and use the template URL if you are using a different region.\
> <https://soracom-files.s3.amazonaws.com/cfn-samples/SoracomAwsCfnSampleFirehoseToS3Stack.yml>

> [!WARNING]
>
> For details on this template, refer to **AWS CloudFormation Resource Stack Details** in the **References** section below.

2. Configure the settings below:

   | Parameter | Content |
   | - | - |
   | **CoverageAWSAccountID** | This parameter depends on your IoT SIM's coverage:<br>- JP Coverage: `762707677580`<br>- Global Coverage: `950858143650` |
   | **DeliveryStreamName** | Enter any stream name |
   | **IamRoleExternalIdForFunnel** | Enter any external id, then save it somewhere since it will be used later (e.g. external-id-0123456789) |
   | **IamRoleNameForFunnel** | Enter any IAM Role name |
   | **S3BucketName** | Enter any S3 Bucket name |

   ![Security](https://docs.soracom.io/_astro/step1-2.B2dxOHds_1aQP5R.webp)

3. Click the check box next to **I acknowledge that AWS CloudFormation...**, then click **Create stack**

   ![Security](https://docs.soracom.io/_astro/step1-3.BZ-8MIv__Z1N3vNK.webp)

> [!CAUTION]
>
> You have successfully set up AWS Resources.

### Retrieve IAM Role ARN

1. Navigate to **[AWS IAM](https://aws.amazon.com/iam/)** Roles

   ![Security](https://docs.soracom.io/_astro/step1-4.pfxpwfQL_1KUAOx.webp)

2. Search for and click the IAM Role you just created (enter the IAM Role Name you entered when you created the CloudFormation Resource Stack)

   ![Security](https://docs.soracom.io/_astro/step1-5.xoARUPjf_3jMyw.webp)

3. Copy the Role ARN and save it for later

   ![Security](https://docs.soracom.io/_astro/step1-6.DJfBdsmE_dKdvo.webp)

> [!WARNING]
>
> At this point you should have an External ID and a Role ARN saved somewhere.

## Step 2: Configure Soracom Funnel

Now that you have the AWS Resources set up, you will set up and enable Soracom Funnel.

### Register Your AWS IAM Role Credentials

1. Sign in to your **[Soracom console](https://console.soracom.io/)**, then click the top right corner

   ![Security](https://docs.soracom.io/_astro/step2-1.BPU81dd1_2qU8Fe.webp)

2. Click **Security**

   ![Credentials](https://docs.soracom.io/_astro/step2-2.CQLb0_Yj_ZAphRM.webp)

3. Click the **Credentials** tab, then click the **Register credentials** button

   ![Credentials](https://docs.soracom.io/_astro/step2-3.BYi6SRJK_1fjUlO.webp)

4. Enter your IAM credentials as follows, then click **Register**:

   | Parameter | Content |
   | - | - |
   | **Credential set ID** | Any name for this credential set (i.e. AWS-IAM-Role-Soracom-Funnel-s3) |
   | **Type** | AWS IAM Role |
   | **Role ARN** | Role ARN of the IAM Role you created in Step 1 |
   | **External ID** | Your IAM Role's external id from Step 1 |

   ![Credentials](https://docs.soracom.io/_astro/step2-4.CRRtG3EL_Z15aqQ7.webp)

> [!WARNING]
>
> For a more detailed guide on this section, go to [Creating a Credential Set](https://docs.soracom.io/en/services/authentication/credential-sets), while still using the specifications above.

### Enable Soracom Funnel

1. Add all desired SIMs to a group, then click **[SORACOM Funnel](https://docs.soracom.io/en/services/funnel)**.

   ![Credentials](https://docs.soracom.io/_astro/step2-5.DqpuO_bQ_Z1CS5k3.webp)

> [!WARNING]
>
> Setting up Soracom Funnel is done at the group level. For more information on Groups and how to add your IoT SIMs to a group, check the Groups section in the [User-Console Guide](https://docs.soracom.io/en/guides/user-console).

2. Turn **Soracom Funnel** on by clicking the switch

3. Configure all the below settings for Funnel

   | Parameter | Content |
   | - | - |
   | **Service** | Amazon Data Firehose |
   | **Destination** | [https://firehose](https://firehose/).\<Your Region code>.amazonaws.com/\<DeliveryStreamName> (e.g. <https://firehose.ap-northeast-1.amazonaws.com/funnel-delivery-stream>) |
   | **Credentials** | The credentials you just created |
   | **Content Type** | JSON |

   ![Credentials](https://docs.soracom.io/_astro/step2-6.BVrqHcBO_ZJkdII.webp)

4. Click **Save**

   ![Credentials](https://docs.soracom.io/_astro/step2-7.C8rfDldv_ohkAC.webp)

> [!CAUTION]
>
> You have successfully set up and activated Soracom Funnel.

## Step 3: Send Data to Soracom Funnel

Now that you have all the set up done, send a request to Funnel's entry point.

We have provided the following examples for connecting with TCP, UDP, and HTTP.

**TCP**

```bash
nc funnel.soracom.io 23080
{"temperature":20} [Enter]

200
[Ctrl+C]
```

**UDP**

```bash
nc -u funnel.soracom.io 23080
{"temperature":20} [Enter]

200
[Ctrl+C]
```

**HTTP**

```bash
curl -vX POST http://funnel.soracom.io \
  -d '{
        "temperature": 20
      }' \
  -H "Content-Type: application/json"

* Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 100.127.65.43:80...
* Connected to funnel.soracom.io (100.127.65.43) port 80 (#0)
> POST / HTTP/1.1
> Host: funnel.soracom.io
> User-Agent: curl/7.74.0
> Accept: */*
> Content-Type:application/json
> Content-Length: 18
>
* upload completely sent off: 18 out of 18 bytes
< HTTP/1.1 204 No Content
< Date: Fri, 28 Oct 2022 12:54:03 GMT
< Connection: keep-alive
< Keep-Alive: timeout=5
<
* Connection #0 to host funnel.soracom.io left intact
```

## Step 4: Confirm Your Data in S3

Now that you've sent data through Soracom Funnel, let's actually confirm that the data arrived in S3.

1. Open and sign in to the **[AWS S3 Console](https://us-east-1.console.aws.amazon.com/s3/buckets?region=us-east-1)**

2. Search for and click the S3 Bucket you created in Step 1

   ![Credentials](https://docs.soracom.io/_astro/step4-1.BaVTo2HP_1TWAp3.webp)

3. Click the latest year, month, day, hour folders

   ![Credentials](https://docs.soracom.io/_astro/step4-2.DwyZW_Nr_s96eC.webp)

4. Click the newly created **Object**

   ![Credentials](https://docs.soracom.io/_astro/step4-3.DhFkX4g4_Z1XFICT.webp)

5. Click **Download**

   ![Credentials](https://docs.soracom.io/_astro/step4-4.DmzYijX__ajKYW.webp)

6. Open the file and confirm that the data you provided through Soracom Funnel is in the file

   ```
   {"operatorId": "OP00XXXXXXXX", "timestamp": 1666961643739, "destination": {"provider": "aws", "service": "firehose", "resourceUrl": "https://firehose.ap-northeast-1.amazonaws.com/XXXXXXXX", "payloadsOnly": false}, "credentialsId": "XXXXXXXX", "payloads": {"temperature": 20}, "sourceProtocol": "tcp", "imsi": "440XXXXXXXXXXXX", "imei": "XXXXXXXXXXXXXXX"}
   {"operatorId": "OP00XXXXXXXX", "timestamp": 1666961643739, "destination": {"provider": "aws", "service": "firehose", "resourceUrl": "https://firehose.ap-northeast-1.amazonaws.com/XXXXXXXX", "payloadsOnly": false}, "credentialsId": "XXXXXXXX", "payloads": {"temperature": 20}, "sourceProtocol": "udp", "imsi": "440XXXXXXXXXXXX", "imei": "XXXXXXXXXXXXXXX"}
   {"operatorId": "OP00XXXXXXXX", "timestamp": 1666961643739, "destination": {"provider": "aws", "service": "firehose", "resourceUrl": "https://firehose.ap-northeast-1.amazonaws.com/XXXXXXXX", "payloadsOnly": false}, "credentialsId": "XXXXXXXX", "payloads": {"temperature": 20}, "sourceProtocol": "http", "imsi": "440XXXXXXXXXXXX", "imei": "XXXXXXXXXXXXXXX"}
   ```

> [!CAUTION]
>
> You have successfully stored data in S3 through Soracom Funnel and Amazon Data Firehose.

## References

### AWS CloudFormation Resource Stack Details

Below are the details of the CloudFormation Resource Stack you created in Step 1.

| AWS Resource | Resource Name |
| - | - |
| **Amazon Data Firehose** | DeliveryStreamName from Step 1 |
| **S3 Bucket** | S3BucketName from Step 1 |
| **IAM Role** | IamRoleNameForFunnel from Step 1 |
| **IAM Role** | \<DeliveryStreamName>-role |
| **CloudWatch Logs Log Group** | \<DeliveryStreamName>-log-group |
| **CloudWatch Logs Log Stream** | \<DeliveryStreamName>-log-stream |

### Terminate Usage

When you want to stop using Funnel to store your data in S3, disable the Soracom Funnel service you enabled in Step 2, then simply delete the CloudFormation Resource Stack you made in Step 1.

1. Navigate to the Amazon CloudFormation Resource Stack you created in Step 1

2. Click **Delete**

   ![Credentials](https://docs.soracom.io/_astro/ref-1.q4kMHu78_1WiejI.webp)

> [!NOTE]
>
> The S3 bucket will not terminate even if you delete the Stack.\
> Visit the Amazon guide on **[Deleting a Bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/delete-bucket.html)** to delete the S3 Bucket resource.