Skip to main content

API Indexing

Whether you want to avoid rate limits, increase data fetching speeds or protect your applications from service outages, TakeShape's API Indexing is a great solution.

This guide will walk you through configuring your TakeShape project to index data from your eCommerce APIs on a schedule, or when webhooks are triggered. Advanced users should read our Spec Reference to get started much faster.

Follow Along

Follow along with this guide by checking out our starter project repo. Our starters let you instantly deploy a pre-configured TakeShape project.

Setting up your project​

To use TakeShape's API Indexing, you'll need a project with a connected service and a project schema with a valid indexedShapes object.

If you're new, check out our guide for creating projects. We also have a guide on connecting services, and a list of guides for many major services, like Shopify or BigCommerce.

Using the UI​

After connecting your Shopify service, you'll immediatey see a popup modal where you can switch on API Indexing for any of your Shopify data. There's a filter field above the list of indexable data, where you can type out the name of the shape you'd like to index. Type in product or collection to see one of the examples we've set up in our Shopify API Indexing starter pattern. Once you've activated the data you'd like to index, select "Update Schema."

NOTE

The Shopify Admin API works in TakeShape via TakeShape's Shopify app, which is automatically installed in your store when you add it to your Shopify service in your project. The TakeShape app has permissions to read orders, products, collections, and customers. Other data may be impossible to index. Test out what queries you can call from TakeShape by running them in your API Explorer.

To manually open the API Indexing menu, select the three dots beside the service you want to index in the Services list on the left side. In the menu that appears, select Configure Indexing.

You can search for your indexed data with the search field in the navbar at the top-right of the web client. Enter the name of any product, collection or other data you've indexed, and it'll show up in the results.

Alternatively, you can navigate to the Data tab and see your indexed data of any shape by selecting it in the sidebar on the left.

If you can't see your indexed data, try triggering reindexing by navigating to the service page of the API you're indexing data from. For example, below is a screenshot of the Reindexing button on the Shopify service page.

To fetch your indexed data from a frontend application, you'll need to create a query that fetches it from ShapeDB.

Configuring your indexedShapes​

Your schema's indexedShapes is a root-level object that you can configure to specify what data TakeShape's indexing function will cache, and how it will do so. Your indexed data must conform to a shape already defined in your schema, and your indexedShapes object's properties will be the names of the shapes you want to index.

For example, your indexedShapes may look like this:

"indexedShapes": {
"Stripe_Customer": {
"queries": {
"list": {
"name": "Stripe_listCustomers"
}
},
"triggers": [
{
"type": "schedule",
"query": "list",
"interval": 1440
},
]
}
}

Each shape contained in your indexedShapes will have two properties: queries and triggers. The queries object contains configuration information for the get and list queries that will be used to fetch data for the index. The triggers array contains configuration information for how these queries will be triggered.

Now let's explore the process of configuring an indexed shape.

Using service shapes​

If you don't have shapes for your service's data defined in your schema, navigate to the API tab in your project's dashboard. There, you can select the service you wish to import shape data for. You'll be taken to the service page, where you can select the Import Data button to begin importing your data.

Finally, a popup modal will be shown, which will allow you to choose which data you'd like to import from your project. You can also indicate whether or not you'd like to create new shapes in your schema to represent that data. If you create a new shape, you can define the name of that shape here as well.

Using service queries​

In the below example, you'll notice the use of Shopify_products for the name of the list query. When the indexer runs for this shape, it will execute the Shopify_products query to fetch a list of Shopify_Product. Just as the Shopify_Product shape listed in the indexedShapes must be a shape that is already defined in your schema, Shopify_products is a query that must be defined in your schema as well.

"indexedShapes": {
"Shopify_Product": {
"queries": {
"list": {
"name": "Shopify_products"
}
},
"triggers": [
{
"type": "schedule",
"query": "list",
"interval": 1440
},
]
}
}

You can manually create the list query for your service, but it's easier to let TakeShape generate the query for you.

To do so, visit the service page as demonstrated above, where you'll find the "Select Queries/Mutations" button. Selecting this button will open a popup modal where you can choose to import generated queries for your service into your schema.

Configuring queries in your indexed shape​

There are two types of queries: list and get. You configure list queries to index a paginated list of the corresponding shape in your service, and configure get to index a single instance of that shape. You can read up on the appropriate properties for configuring these queries in the schema spec reference.

Here are some examples of valid list and get configurations:

Shopify:


"Shopify_Collection": {
"queries": {
"list": {
"name": "Shopify_collections",
"ignoreFields": [
"availablePublicationCount",
"publicationCount",
"publishedOnCurrentPublication",
"productPublications",
"publications",
"resourcePublications",
"resourcePublicationsV2",
"unpublishedChannels",
"unpublishedPublications",
],
"objectDepthLimit": 1
},
"get": {
"name": "Shopify_collection",
"ignoreFields": [
"availablePublicationCount",
"publicationCount",
"publishedOnCurrentPublication",
"productPublications",
"publications",
"resourcePublications",
"resourcePublicationsV2",
"unpublishedChannels",
"unpublishedPublications",
],
"objectDepthLimit": 1
}
},
"triggers": [
{"type": "schedule", "query": "list", "interval": 1440},
{"type": "schedule", "query": "get", "interval": 1440}
]
}

Note the ignoreFields in the above example. The indexer will run your queries with every valid field possible in the selectionset. To exclude a field from the selectionset, create an ignoreFields array in your query.

Configuring pagination is not necessary for Shopify because Shopify's API implements Relay's cursor-based connections specification.

TakeShape will automatically paginate through data returned from list queries to GraphQL APIs that implement Relay-style pagination. If your service's documentation doesn't explicitly state whether or not it uses Relay-style pagination, you can usually tell by how list queries are structured.

List queries for APIs that use Relay-style pagination require the use of edges and nodes to access the data in each item of the list. For example, below is a query that lists all Shopify products from the Admin API:

Shopify Products List Query
Shopify_products(first:10){
edges{
node{
id
title
}
}
}

[Shopify uses Relay pagination]((https://www.shopify.com/ph/partners/blog/graphql-pagination), thus no extra pagination configuation is required.

NOTE

Relay is Facebook's GraphQL client, and you can read more about this particular style of cursor-based pagination in the GraphQL docs.

Stripe:

"indexedShapes": {
"Stripe_Customer": {
"queries": {
"list": {
"name": "Stripe_listCustomers"
}
},
"triggers": [
{"type": "schedule", "query": "list", "interval": 1440},
{"type": "webhook", "query": "list", "service": "stripe", "events": ["customer.created", "customer.updated", "customer.deleted"]},
]
}
}

TakeShape's indexer paginates Stripe data by default. There's no need to configure pagination.

When setting up your Stripe service you can also configure webhooks for more responsive indexing. The steps involved are:

  1. Add your Stripe service.
  2. Set up an indexed shape as in the above example.
  3. In your Stripe dashboard go to Developers > Webhooks and create a new webhook using an endpoint that follows this scheme: https://api.takeshape.io/project/{YOUR_PROJECT_ID}/webhooks/indexing/{YOUR_STRIPE_SERVICE_ID}
  4. Add a trigger to the triggers array above that looks like this. Notice the events which should match the events you which to intercept to refresh your index. {"type": "webhook", "query": "list", "service": "{YOUR_STRIPE_SERVICE_ID}", "events": ["customer.created", "customer.updated", "customer.deleted"]}
  5. Copy the webhook secret from your new Stripe webhook. Paste it into the Stripe service config on TakeShape in the Webhook Secret field.

Configuring triggers in your indexed shape​

There are two valid types of triggers: webhook and schedule. Use your triggers to configure when the API indexer will cache data fetched from your defined queries. Let's look at two examples of trigger configs:

BigCommerce:

"indexedShapes": {
"BigCommerce_Product": {
"queries": {
"get": {
"name": "BigCommerce_product"
},
"list": {
"name": "BigCommerce_products"
}
},
"triggers": [
{
"type": "webhook",
"query": "get",
"service": "bigcommerce",
"events": ["store/product/created"]
},
{
"type": "schedule",
"query": "list",
"service": "bigcommerce",
"interval": 1440
}
]
}

Webhook triggers:

Note the use of the webhook trigger type in the example above. Shapes with queries triggered by webhooks will only begin indexing when the events you specify happen. You can find a list of webhook events for your service in the service's documentation. For example, BigCommerce lists all of its events.

Schedule triggers:

There's also a schedule trigger defined in the above example, and you'll notice that the interval is set to 1440. The interval property takes an integer number of minutes that TakeShape will wait between attempts at indexing your data. We recommend setting your interval to no less than 1440 in most instances, as that will cause your indexer to run every 24 hours.

Generic REST

"indexedShapes": {
"Recharge_Product": {
"queries": {
"list": {
"name": "listRechargeProducts",
"pagination": {
"type": "cursor",
"cursorPath": "products[(@.length-1)].id",
"itemsToIndexPath": "products",
"hasMorePath": "next_cursor",
"cursorArg": "next_cursor",
"pageSizeArg": "limit"
},
"objectDepthLimit": 2
}
},
"triggers": [{"type": "schedule", "query": "list", "interval": 1440}]
}
}

There are two important factors when indexing data from a generic rest service: Configuring pagination and creating shapes for the data.

The above example uses Recharge, a service that enables subscriptions in Shopify stores. You can find pagination specifications for your REST service in its docs, like this guide to Recharge pagination.

NOTE

Some APIs have multiple versions, and multiple corresponding versions of their documentation. For example, Recharge's 2021-01 API uses page-based pagination, but their 2021-11 uses cursor-based pagination.

You can learn which properties are valid for configuring your pagination object in the schema spec reference.

The data returned from the queries specified in your indexed shape must match a shape defined in your schema. Read our guide on working with shapes to learn how to create a shape for your generic rest service's data.

Creating queries for your API index​

Finally, to use this indexed data in your application outside of TakeShape, you must create a query in your schema to fetch your data from the index. You can follow our guide on creating queries and mutations to do so. Below is an example of a query that fetches indexed Shopify products:

"getIndexedProductList": {
"shape": "PaginatedList<Shopify_Product>",
"resolver": {
"shapeName": "Shopify_Product",
"name": "takeshape:queryApiIndex",
"service": "takeshape:local",
"options": {"indexedShape": "Shopify_Product"}
},
"description": "Fetch Shopify products from the API Index.",
"args": "TSListArgs<Shopify_Product>"
},

In the above example, note that the relevant shape in the indexedShapes object is specified. If this is not correctly configured, the query will not work. Note that the name property for the resolver is also set to takeshape:queryApiIndex. You can learn more about configuring resolvers in our resolver spec.

Another important point for creating a query that fetches from your project's index is that the query's shape must match that of the data in the index. If you want to use TakeShape's generated shapes for your provided service, read our section above on using your service's shapes.

Troubleshooting your API index​

API Indexing will trigger whenever you deploy a schema with changes to the triggers of one of your indexed shapes, or import data from a service into your project. When you're on your project dashboard in TakeShape's web client, you'll see a small popup modal indicating the beginning and end of indexing.

If indexing any of your shapes fails, there are a few areas you can examine immediately for quick potential fixes.

First steps​

If you're having issues with your API Indexing, first check that you've configured your indexedShapes correctly. Confirm that your configuration matches the valid properties for the service you're using. If you're indexing a list query, confirm that your pagination is accurately configured to query your service. Below is a list of links to the pagination documentation of some of the most common eCommerce services:

See the examples above for more help configuring your queries.

Check the API Indexing log​

In your project's dashboard in the TakeShape web client, open up the API Indexing Log by selecting its button. Navigate to the API Tab, then select the button beside the Export Schema buttonβ€”its icon looks like a bulleted list.

You'll see a filterable list of API Indexing events. Select the Inspect Payload button beside failed items to see what query the indexer ran and what result was returned. The button looks like a small i icon.

If there are issues with the query the indexer is running, confirm that the query you specified for fetching your indexed shape returns the data you're expecting.

Check your ignoreFields​

The ignoreFields array in your query object specifies which fields to ignore when the indexer queries your service. The indexer will add every possible field to the selectionset when querying your service, and some services have query fields only available to users at higher subscription tiers. If your account with the service is not allowed to query those fields, you must include them in your ignoreFields array to successfully query the service. Here is an example of an ignoreFields array that instructs the indexer to omit fields:

"ignoreFields": [
"fieldExample1",
"fieldExample2",
"fieldExample3",
]
Shopify users are covered

TakeShape's indexer automatically removes premium fields from the selectionset when querying the Shopify Admin API. Shopify users don't need to specify an ignoreFields array.

Check your objectDepthLimit​

If your objectDepthLimit is too high, this may cause the query sent to the service to be too complex, which could result in the API Indexer receiving an error response. For example, the proper objectDepthLimit for Shopify is 1. Experiment to see what works for your service.

Check your triggers​

Webhook triggers can be troublesome if your service's documentation is unclear. Be sure that the event names you provide to the events array in your triggers object are valid. Ensure also that you are referencing documentation relevant to the version of the API you are using, as there can be major differences between API versions for eCommerce service providers.

Here is an example of a webhook trigger for a shopify service:

{
"type": "webhook",
"query": "get",
"service": "my-shopify-shop",
"events": ["products/create", "products/update", "products/delete"]
}

Below is a list of links to the webhook events for some of the most popular eCommerce services:

Wildcards​

We don't currently support wildcards (the * symbol) in webhook event handling, so a trigger with the follow events array will not index:

"events": ["store/carts/*"]

Still need help? Get in touch with us.