You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Sat-api is a STAC compliant web API for searching and serving satellite imagery metadata. Development Seed runs an instance of it for Landsat and Sentinel imagery hosted on AWS. You can access this instance at https://sat-api.developmentseed.org.
6
6
7
-
v1.0: This API uses Elastic Search as its engine and uses on AWS's Lambda and APIGateway to update and serve the data. A live API of the master branch is deployed tohttps://sat-api.developmentseed.org
7
+
An older version of sat-api can be found on the [legacy branch ](https://github.com/sat-utils/sat-api/tree/legacy)and is deployed athttps://api.developmentseed.org/satellites.
8
8
9
-
v0.3.0 (sat-api legacy): A sat-api legacy version can be found on the [legacy branch ](https://github.com/sat-utils/sat-api/tree/legacy) and is deployed at https://api.developmentseed.org/satellites.
9
+
This repo includes a number of npm packages that are used to create and populate an instance of sat-api. For the full list of packages go to:
10
+
https://www.npmjs.com/org/sat-utils
10
11
11
-
## SpatioTemporal Asset Catalog (STAC) compliant
12
+
## Documentation
12
13
13
-
A STAC is made up of `items`, which is data for a specific time and place. An `item` may contain multiple files (e.g., bands, quality data), but covers an identical geographical footprint for specific date and time. There are only a few so it is worth listing what the core fields are:
14
-
15
-
-`id`: A unique ID assigned by the data provider for this item.
16
-
-`geometry`: Polygon geojson geometry describing the area covered by the data
17
-
-`datetime`: The date and time of the item
18
-
-`provider`: Although an optional field, this is the name of the person or organization providing the data.
19
-
-`license`: The name of the license regarding data use.
20
-
-`assets`: This contains the individual resources that are accessed via a URI. Typically a web URL, this could also be a location on s3 of Google
21
-
-`links`: links are not the actual data, but any weblinks associated with the data. A 'self' link is required.
22
-
23
-
#### STAC extensions
24
-
25
-
The STAC specification is designed to be extended for different types of geospatial data, each of which may have different set of important metadata fields. LiDAR, Video, Mosaics, and Earth Observation imagery are all different types of data that could may eventually have their own extension.
26
-
27
-
The most common and immediate use case is what we call Earth Observeration data. This refers to raster imagery taken from a sensor at discreate date and time with multiple bands from the visible through long wave infrared. This may commonly be called a granule, or a scene, but in STAC it is referred to as an `item`.
28
-
29
-
Rather then get into the specifics of the EO spec, which can be found in the [stac-spec GitHub repository](https://github.com/radiantearth/stac-spec/blob/master/extensions/stac-eo-spec.md) I will instead use examples from sat-api.
30
-
31
-
32
-
## Searching sat-api
33
-
34
-
Now with an undersanding of how STAC items and collections work we can look at how to query data. Behind the scenes sat-api queries both collections and items to find matches, so that if you search for `eo:off_nadir=0`, it will return all of the Landsat-8 scenes even though `eo:off_nadir` doesn't appear in the `items` themselves, only in the landsat-8 collection.
35
-
36
-
Any metadata field that appears in the `items` or `collection` properties can be queried by providing those as parameters in the form of key=value as shown below.
No search term is more important however than the a geospatial query to find data covering a specific area. The core STAC spec allows for searching by providing a bounding box, with more complex 'intersects' query to query against user provided polygons. Sat-api does not currently support the [simpler] bounding box query, but does support the 'intersects' query.
57
-
58
-
**Caveat emptor**: Due to the way sat-api does the two pronged search of collections and items, a side effect is that queries fields that are absent from both will match scenes. For example, if a typo occurs in your query like `datetme=2017-08`, then that parameter will be ignored but no warning will be issued. This problem will be fixed once we implement validators into the API.
59
-
60
-
## Deploy your own sat-api
61
-
62
-
Work is ongoing to simplify the deployment process, but the following steps should get you started:
63
-
64
-
1. Make sure you have AWS credentials with necessary access to AWS Lambda and AWS APIGateway (an admin level access will do enough).
65
-
66
-
2. To simplify deployment to AWS, we make use of [`kes`](http://devseed.com/kes/), a tool for CloudFormation deployment. It can be installed with:
67
-
68
-
$ npm install -g kes
69
-
70
-
3. You MUST create a bucket on S3 that is used for storing deployment artifacts and metadata csv files.
71
-
72
-
4. Update `.kes/config.yml` and enter the name of the bucket.
73
-
74
-
5. If direct access to the elasticsearch instance is needed from fixed IP address, copy `.env.example` to `.env` and add your IP address there.
75
-
76
-
6. Deploy the system with `kes`. It can take some time for AWS to create the various resources, so be patient.
77
-
78
-
$ kes cf deploy -r us-east-1
79
-
80
-
Replace `us-east-1` with any desired region. This will deploy the CloudFormation stack, which includes:
81
-
82
-
* API Gateway
83
-
* Lambda functions
84
-
* Step Functions
85
-
* CloudWatch Rules
86
-
* Elasticsearch
87
-
* Associated IAM Roles
88
-
89
-
Additional calls of `kes cf deploy` will update the existing CloudFormation stack (see below on building the deployment distribution).
90
-
91
-
7. Once the system is initialized, go to the API Gateway console and select the "sat-api-..." entry, click on the _Settings_ menu, and then click _Add Binary Media Type_ option and add `'*'` as the Binary media type.
92
-
93
-
The Landsat and Sentinel ingestors are run as Step Functions every 12 hours (Landsat at 6:00 and 18:00, Sentinel at 0:00 and 12:00), as can be seen under the CloudWatch Rules console. They can be disabled from the console.
94
-
95
-
### Elasticsearch Management
96
-
97
-
A Lambda function is included that provides Elasticsearch management functions from within the AWS Lambda Console. Navigate to the Lambda functions console for the region the sat-api stack has been deployed to and locate the *stackName*-manager Lambda function. From here you can configure test events that consist of a JSON payload.
98
-
99
-
```
100
-
{
101
-
"action": action,
102
-
"index": "items" or "collections"
103
-
}
104
-
```
105
-
106
-
Unless it has been changed in the code, the main index used in the Elasticsearch instance will always be sat-api. The action parameter can be:
107
-
108
-
-`putMapping`: Puts a new mapping for indexing. This is done automatically during ingestion if it does not already exist so should never need to be used directly.
109
-
-`deleteIndex`: This deletes the index. Use with caution!
110
-
-`listIndices`: This returns a list of all indices. Unless some have been added should include 'items', 'collections' and '.kibana'
111
-
-`reindex`: Spawns a reindexing of all records
14
+
Go [here](docs/README.md)
112
15
113
16
## Development
114
17
115
-
A live API of the develop branch is auto-deployed to https://sat-api.developmentseed.org
116
-
117
-
To further develop the API, install dependenceis with yarn and build the files for deployment (using webpack). This creates moduels under dist/ that can be deployed as the Lambda function source code.
18
+
### Local Installation
118
19
119
20
$ yarn
120
21
$ yarn bootstrap
@@ -123,7 +24,9 @@ To further develop the API, install dependenceis with yarn and build the files f
123
24
# to continually watch and build source files
124
25
$ yarn watch
125
26
126
-
As noted above, running `kes cf deploy ...` will update the CloudFormation deployment with the latest build.
Sat-api is a STAC compliant web API for searching and serving satellite imagery metadata. Development Seed runs an instance of it for Landsat and Sentinel imagery hosted on AWS. You can access this instance at https://sat-api.developmentseed.org.
6
+
7
+
An older version of sat-api can be found on the [legacy branch](https://github.com/sat-utils/sat-api/tree/legacy) and is deployed at https://api.developmentseed.org/satellites.
8
+
9
+
This repo includes a number of npm packages that are used to create and populate an instance of sat-api. For the full list of packages go to:
Now with an undersanding of how STAC items and collections work we can look at how to query data. Behind the scenes sat-api queries both collections and items to find matches, so that if you search for `eo:off_nadir=0`, it will return all of the Landsat-8 scenes even though `eo:off_nadir` doesn't appear in the `items` themselves, only in the landsat-8 collection.
4
+
5
+
Any metadata field that appears in the `items` or `collection` properties can be queried by providing those as parameters in the form of key=value as shown below.
No search term is more important however than the a geospatial query to find data covering a specific area. The core STAC spec allows for searching by providing a bounding box, with more complex 'intersects' query to query against user provided polygons. Sat-api does not currently support the [simpler] bounding box query, but does support the 'intersects' query.
26
+
27
+
**Caveat emptor**: Due to the way sat-api does the two pronged search of collections and items, a side effect is that queries fields that are absent from both will match scenes. For example, if a typo occurs in your query like `datetme=2017-08`, then that parameter will be ignored but no warning will be issued. This problem will be fixed once we implement validators into the API.
Work is ongoing to simplify the deployment process, but the following steps should get you started:
4
+
5
+
1. Make sure you have AWS credentials with necessary access to AWS Lambda and AWS APIGateway (an admin level access will do enough).
6
+
7
+
2. To simplify deployment to AWS, we make use of [`kes`](http://devseed.com/kes/), a tool for CloudFormation deployment. It can be installed with:
8
+
9
+
$ npm install -g kes
10
+
11
+
3. You MUST create a bucket on S3 that is used for storing deployment artifacts and metadata csv files.
12
+
13
+
4. Update `.kes/config.yml` and enter the name of the bucket.
14
+
15
+
5. If direct access to the elasticsearch instance is needed from fixed IP address, copy `.env.example` to `.env` and add your IP address there.
16
+
17
+
6. Deploy the system with `kes`. It can take some time for AWS to create the various resources, so be patient.
18
+
19
+
$ kes cf deploy -r us-east-1
20
+
21
+
Replace `us-east-1` with any desired region. This will deploy the CloudFormation stack, which includes:
22
+
23
+
* API Gateway
24
+
* Lambda functions
25
+
* Step Functions
26
+
* CloudWatch Rules
27
+
* Elasticsearch
28
+
* Associated IAM Roles
29
+
30
+
Additional calls of `kes cf deploy` will update the existing CloudFormation stack (see below on building the deployment distribution).
31
+
32
+
7. Once the system is initialized, go to the API Gateway console and select the "sat-api-..." entry, click on the _Settings_ menu, and then click _Add Binary Media Type_ option and add `'*'` as the Binary media type.
33
+
34
+
The Landsat and Sentinel ingestors are run as Step Functions every 12 hours (Landsat at 6:00 and 18:00, Sentinel at 0:00 and 12:00), as can be seen under the CloudWatch Rules console. They can be disabled from the console.
A Lambda function is included that provides Elasticsearch management functions from within the AWS Lambda Console. Navigate to the Lambda functions console for the region the sat-api stack has been deployed to and locate the *stackName*-manager Lambda function. From here you can configure test events that consist of a JSON payload.
4
+
5
+
```
6
+
{
7
+
"action": action,
8
+
"index": "items" or "collections"
9
+
}
10
+
```
11
+
12
+
Unless it has been changed in the code, the main index used in the Elasticsearch instance will always be sat-api. The action parameter can be:
13
+
14
+
-`putMapping`: Puts a new mapping for indexing. This is done automatically during ingestion if it does not already exist so should never need to be used directly.
15
+
-`deleteIndex`: This deletes the index. Use with caution!
16
+
-`listIndices`: This returns a list of all indices. Unless some have been added should include 'items', 'collections' and '.kibana'
A STAC is made up of `items`, which is data for a specific time and place. An `item` may contain multiple files (e.g., bands, quality data), but covers an identical geographical footprint for specific date and time. There are only a few so it is worth listing what the core fields are:
6
+
7
+
-`id`: A unique ID assigned by the data provider for this item.
8
+
-`geometry`: Polygon geojson geometry describing the area covered by the data
9
+
-`datetime`: The date and time of the item
10
+
-`provider`: Although an optional field, this is the name of the person or organization providing the data.
11
+
-`license`: The name of the license regarding data use.
12
+
-`assets`: This contains the individual resources that are accessed via a URI. Typically a web URL, this could also be a location on s3 of Google
13
+
-`links`: links are not the actual data, but any weblinks associated with the data. A 'self' link is required.
14
+
15
+
#### STAC extensions
16
+
17
+
The STAC specification is designed to be extended for different types of geospatial data, each of which may have different set of important metadata fields. LiDAR, Video, Mosaics, and Earth Observation imagery are all different types of data that could may eventually have their own extension.
18
+
19
+
The most common and immediate use case is what we call Earth Observeration data. This refers to raster imagery taken from a sensor at discreate date and time with multiple bands from the visible through long wave infrared. This may commonly be called a granule, or a scene, but in STAC it is referred to as an `item`.
20
+
21
+
Rather then get into the specifics of the EO spec, which can be found in the [stac-spec GitHub repository](https://github.com/radiantearth/stac-spec/blob/master/extensions/stac-eo-spec.md) I will instead use examples from sat-api.
0 commit comments