Consent

Cognite Hub is going public. So far, Cognite Hub has required users to log in to access some of its content. Now we have decided to open most parts of the community up to the public. For more information, please read our Privacy Policy and Terms and Conditions.

If you don’t agree with this privacy statement, you can reach out to support@cognite.com

×

COGNITE HUB

Check the documentation

Ask the Community

Take a look at Academy

Cognite Status Page

Contact Cognite Support

Products Cognite Data Fusion

Cognite InField

Cognite InRobot

Cognite Maintain

Cognite Remote

Website cognite.com

Groups
- - Knowledge Base
Product Updates
Product Ideas
Docs
Events
Academy
Our Partners

Impact 2024: The Industrial Data and AI Conference for and by Users | Nominate Speakers Now for a Ch...

(Mon, 14 Oct, 2:00 pm)

Home
Search

Topics started by Pierre Pernot

19 Topics

Recently active

Most views Most replies

Pierre PernotPractitioner

Transformations: Asset External IDs instead of IDs for linking Events, Timeseries, Sequences to AssetsGathering Interest

Hello, That would be nice to be able to use the asset external IDs (instead of the assetId field) in transformations, to link events (or timeseries, sequences) to existing assets.

Pierre PernotPractitioner

posted in How-To Guides

How to use the MQTT hosted extractor [Cognite Official]

MQTT is one of the standards for IoT communication in the industry. It is a lightweight, publish-subscribe, machine to machine network protocol for message queue/message queuing service. As it is widely used, it made sense to have a dedicated extractor for that protocol. At Cognite, we have a hosted extractor for MQTT, that facilitates the extraction of data from a broker. “Hosted” means it runs on our infrastructure and you don’t have to deploy anything on yours to extract data: you just have to configure it. In this article, we will see how to configure and use it. An example scenario involving MQTT could be sensors installed on industrial machines publishing timeseries data like temperature, pression etc., to a broker. Data would be published to different topics. When extracting data from the broker, with the Cognite extractor, we can have different behaviors based on the topic. Here, we will publish data to four different topics, on an MQTT broker. Those topics are named as follow

Pierre PernotPractitioner

Ability to download a group manifestPlanned for development

Hello,When using CDF the toolkit, for example, we need to write group capabilities as JSON.To retrieve the group capabilities in a JSON format, we currently have to go to Developer Tools > Network, select the right request and get the JSON.It would be nice if we could just download the group manifest (like we can do for transformations) directly in the UI.

Pierre PernotPractitioner

posted in Developer and User Community

How to use contextualization with the SDK

Contextualization is one of CDF’s core strengths. It allows you to link resources from different data sources together, which can be pretty powerful in an industrial context. We can for example think about linking an asset hierarchy based on SAP, to some time series extracted from sensors. You can contextualize CDF resources in the UI. In case you need more filtering capabilities, or you want to automate that, you can also do that programmatically. In this article, we’ll take the example of contextualizing files and assets with the SDK. The setup we will use will look like that:20 assets 20 files, for which something similar to the asset name appears in a metadata field called “asset” It is often the case that files and assets don’t share something obvious to make a link on, because they might be ingested from different sources. That is where contextualization comes into play, using Machine Learning to find matches between resources based on specified fields. To create the setup, we

Pierre PernotPractitioner

posted in How-To Guides

How to package a custom extractor as a Docker image [Cognite Official]

Docker is one of the most common ways to deploy an extractor because of its ease of use. Our off-the-shelf extractors have a Docker image you can download and run. But when it comes to custom extractors, you have to build the Docker image yourself (if you intend to run it with Docker, of course). In this article, we’ll see how to do that. We’ll use the example of the REST extractor, from a previous article (https://hub.cognite.com/developer-and-user-community-134/rest-extractor-1232). It is a custom extractor, made with our extractor-utils package, hence it does not have a pre-built Docker image.The content of this article will be specific to the REST extractors made with the extractor-utils package, but is still applicable to other kinds of extractors. You simply have to make sure to adapt environment variables, arguments etc. to what your extractor needs. First, make sure that you have Docker installed, up and running. Open the python project with the extractor in your code editor. T

Pierre PernotPractitioner

posted in How-To Guides

How to run the OPC UA extractor with Docker [Cognite Official]

OPC UA is an open source data standard for communication. It is one of the most used in the industry. OPC UA can for example be used to send data from a machine and its sensors to a computer (or to a cloud service such as Cognite Data Fusion).In this article, we’ll see how to extract data from an OPC UA data source to Cognite Data Fusion (CDF). For this purpose, we have an off-the-shelf extractor that we simply need to configure and deploy. This extractor is maintained by Cognite. You can find it in CDF under Integrate > Extract data (where you can also find our other extractors: DB, PI etc., you can also create custom extractors). One of the most common and easiest ways to deploy such an extractor is via a Docker image. The fact that it is platform agnostic makes it pretty easy to deploy: you just need docker installed to make it run. Also, everything you need is packaged in the image, you do not need to install or build anything. The Docker images are available on Docker Hub (htt

Pierre PernotPractitioner

posted in Developer and User Community

How to create events for different assets with transformations

A common setup in CDF (Cognite Data Fusion) is to first build an asset hierarchy, and then to attach events, timeseries etc. to those assets. One way to do it is to use transformations. In this short article, we’ll take events as an example. It works the same way for timeseries and sequences. Let’s assume we already have an asset hierarchy and we want to attach events to it. We’ll suppose that we have a RAW table, from which we want to create events, that looks like this: In transformations, to attach an asset to a created event, you need to specify the corresponding asset ID (assetId in the target schema). Asset IDs being automatically generated, we usually prefer using external IDs because we know what they are made of: an ID from an ERP, a value following a naming convention, etc. Since the asset ID is the expected value for the assetId field in the target schema, we need to retrieve for each asset its ID, based on its external ID. As mentioned in the documentation, we can read f

Pierre PernotPractitioner

posted in How-To Guides

How to test Python code involving SDK client calls [Cognite Official]

Automated tests are considered a good practice in software development. They can apply to every language or use case. Here we will focus on Python and testing code that involves the Python SDK’s CogniteClient object. Below are listed some of the benefits of testing. First, they are a good way to test your code in a fast and repeated manner. With automated tests, you avoid testing manually your code: you can write each test case programmatically and run them on request. When they fail, running them in a debugger really helps understand the code, each object’s structure, and solve the problem quite fast. Then, it helps make the code maintainable. Indeed, when the code needs to be changed, running your tests ensures that the tested code will still work as expected. When a bug is discovered, it is good to create a test case for that bug to avoid having it again when updating the code. It also helps collaborate on the same repository. When several people work on the same code, it might hap

Pierre PernotPractitioner

posted in Digitalization Community

SAP write-backs from Cognite Data Fusion

A use case we encounter at Cognite is writing data back to SAP. This can happen in several contexts where the main goal is to create or update data in SAP (ex: work orders, notifications etc.) based on the analysis of data stored in Cognite Data Fusion. The ability to write back to SAP allows to take decisions without going back and forth between applications, which can save a lot of time. It is also less error prone than manually filling fields in SAP based on what you read in Cognite Data Fusion. This use case, which is all about automating processes, definitely fits Industry 4.0. In addition to that, SAP is one of the most used ERP systems in the industry. As a side note: we are talking today about SAP, but the same would be possible with other ERPs (as long as they have an API we can send requests to). For example, in a maintenance context: when analyzing data of your industrial machinery, you might notice that one of your machines needs maintenance. Instead of going to SAP, looki

Pierre PernotPractitioner

posted in How-To Guides

How to create and run a REST extractor [Cognite Official]

REST APIs are one of the most common ways to expose data. At Cognite we often encounter use cases where we have to extract data from such APIs (for example extracting data from SAP) to Cognite Data Fusion (CDF). Since every REST API is totally custom, we do not have an off-the-shelf REST extractor. Nevertheless, we have a package called extractor-utils enabling you, among others, to create custom extractors for REST APIs: https://cognite-extractor-utils.readthedocs-hosted.com/en/latest/ In this article, we’ll go through an example using Meteorologisk Institutt’s Frost API, exposing meteorological data from Norway. It is a free API, the documentation is available at https://frost.met.no/api.html#/locations . You need to create a user in order to use the data: https://frost.met.no/howto.html . The goal will be to create an asset and a timeseries for each one of the locations. Then we will feed datapoints (for the meteorological reports) inside of those timeseries thanks to an extractor.

Pierre PernotPractitioner

published in Developer and User Community

Cognite Replicator

A common setup that we have at Cognite is having 3 different environments: development, test and production. We have one specific Cognite Data Fusion (CDF) project for each of these environments. This setup allows to develop new features without interfering with production data. Also, it makes the deployment of new features to production much safer, since they are tested before.Sometimes, you still want to have the data you used during your tests, in your production environment. For example, if you extracted some data from sensors and saved them as time series in CDF, you may want to keep this historical data when you move to production. Same goes with files, assets, datasets, functions etc. Another example could be the replication of the data from a full production CDF project, to one of your customer’s project, containing only a subset of the source one.At Cognite, we have a python package (available at https://pypi.org/project/cognite-replicator/) whose purpose is to copy the select

Pierre PernotPractitioner

posted in How-To Guides

How to validate data with pydantic [Cognite Official]

Users are often required to fill in information: YAML, forms, that is then processed by a script. Other developers might also have to send data to an API you created. Nevertheless, there is no guarantee that the user fills in the correct type of information. You might want two things: First, make sure the data you process is of the correct type If it is not, provide the user some meaningful errors Pydantic is a library that helps with data validation. Indeed, thanks to type hints and validators, you can enforce data types, from simple to complex, and apply validation rules to data. It uses python classes, and is quite easy to use. As an example, let’s say we have a machine. This machine has a name, a make, and an ID. We can create a specific class for it. from pydantic import BaseModel class Machine(BaseModel): id: int name: str make: str external_data = { "id": "123", "name": "Fantastic Machine", "make": "Machine Manufacturer",} my_machine = Machine(**external_data) pr

Pierre PernotPractitioner

posted in Digitalization Community

Cognite Data Fusion Tour - Part 4: Scaling

Cognite Data Fusion is a product built to address the challenges of working with industrial data by: Making data available - Liberate their IT, OT, ET and visual data from siloed source systems with our extractor pipelines. This is done reliably and at scale. Making data meaningful - We use AI-powered contextualisation services to create an Industrial Knowledge Graph that delivers trusted, contextualised data Make data valuable - Cognite Data Fusion enables your teams to access this data with the best-of-breed tools of your choice to turn this data into business value Monolith solutions often end up creating vendor lock-in and can even end up creating more data silos within your organisation With trusted, contextualised data available in an industrial knowledge graph, your teams are equipped to scale solutions both in the volume of new solutions and replicating successful solutions across assets, lines, or sites.The videos are based on the “ice cream factory” use case: a use case

Pierre PernotPractitioner

posted in Digitalization Community

Cognite Data Fusion Tour - Part 3

Cognite Data Fusion is a product built to address the challenges of working with industrial data by: Making data available - Liberate their IT, OT, ET and visual data from siloed source systems with our extractor pipelines. This is done reliably and at scale. Making data meaningful - We use AI-powered contextualisation services to create an Industrial Knowledge Graph that delivers trusted, contextualised data Make data valuable - Cognite Data Fusion enables your teams to access this data with the best-of-breed tools of your choice to turn this data into business value Monolith solutions often end up creating vendor lock-in and can even end up creating more data silos within your organisation With trusted, contextualised data available in an industrial knowledge graph, your teams are equipped to scale solutions both in the volume of new solutions and replicating successful solutions across assets, lines, or sites.The videos are based on the “ice cream factory” use case: a use case

Pierre PernotPractitioner

posted in Digitalization Community

Cognite Data Fusion Tour - Part 2

Cognite Data Fusion is a product built to address the challenges of working with industrial data by: Making data available - Liberate their IT, OT, ET and visual data from siloed source systems with our extractor pipelines. This is done reliably and at scale. Making data meaningful - We use AI-powered contextualisation services to create an Industrial Knowledge Graph that delivers trusted, contextualised data Make data valuable - Cognite Data Fusion enables your teams to access this data with the best-of-breed tools of your choice to turn this data into business value Monolith solutions often end up creating vendor lock-in and can even end up creating more data silos within your organisation With trusted, contextualised data available in an industrial knowledge graph, your teams are equipped to scale solutions both in the volume of new solutions and replicating successful solutions across assets, lines, or sites.The videos are based on the “ice cream factory” use case: a use case

Pierre PernotPractitioner

posted in Digitalization Community

Cognite Data Fusion Tour - Part 1

Cognite Data Fusion is a product built to address the challenges of working with industrial data by: Making data available - Liberate their IT, OT, ET and visual data from siloed source systems with our extractor pipelines. This is done reliably and at scale. Making data meaningful - We use AI-powered contextualisation services to create an Industrial Knowledge Graph that delivers trusted, contextualised data Make data valuable - Congite Data Fusion enables your teams to access this data with the best-of-breed tools of your choice to turn this data into business value Monolith solutions often end up creating vendor lock-in and can even end up creating more data silos within your organisation With trusted, contextualized data available in an industrial knowledge graph, your teams are equipped to scale solutions both in the volume of new solutions and replicating successful solutions across assets, lines, or sites.The videos are based on the “ice cream factory” use case: a use case

Pierre PernotPractitioner

posted in Developer and User Community

Interactions with Cognite Data Fusion in an Android app

Today’s topic is about interacting with Cognite Data Fusion (CDF) from Android devices. Android devices can be phones, but also other kind of devices like hands-free devices (with speech commands). Since Kotlin, the Android preferred language, has interoperability with Java, it facilitates the integration of interactions with CDF in your Android apps. You can indeed use our Java SDK in an Android application, which makes it really easy to retrieve and upload data from/to CDF. The Cognite Java SDK is complete and maintained. This can broaden your thoughts/ideas about what to do with CDF. Below a few examples of what can be done : Retrieve information from CDF to display it on your mobile device Upload data to CDF from a mobile device, which can be useful in an industrial context Using specific features, from mobile devices (like a camera for example) to upload particular data to CDF (raw, or to extract features from them with an ML model, thanks to a Cognite function for example)

Pierre PernotPractitioner

posted in Developer and User Community

Cognite Functions

Hello, Today’s topic is about Cognite functions: Cognite functions enable the user to run and schedule arbitrary code to clean, process and do calculations on your data. They are easy to use and have seamless integration with Cognite Data Fusion (CDF), allowing the Cognite Function to interact with CDF directly. The service automatically scales the computing infrastructure to handle a fluctuating number of function calls. They can be triggered on schedule, on click in the UI, and even with the experimental Python SDK, which makes them adaptable to a lot of use cases. We also have a full CI/CD flow for them, which allows to deploy and maintain them really easily. Some examples of use cases that can be done with Cognite functions:Change point detection, thanks to machine learning models, that run on schedule Anonymisation of videos, with computer vision models @Gaetan Helness is quite familiar with Cognite Functions. If you have any questions, don't hesitate to reach out to us in t

Pierre PernotPractitioner

posted in How-To Guides

How to make API calls with the Python SDK [Cognite Official]

Hello,I’m Pierre, data engineer at Cognite. I’ll be posting, on a regular basis, some useful tips about Cognite Data Fusion and its use. If you have any questions or remarks (technical or not), don’t hesitate to reach out to me or to create a new topic in Cognite Hub. Some of the features of the Cognite API are not available in the Python SDK. You can still use those features by making a call to the desired endpoint. The SDK facilitates this process thanks to the get, post, put and delete methods of the Cognite client (see: https://cognite-docs.readthedocs-hosted.com/projects/cognite-sdk-python/en/latest/cognite.html#cogniteclient). Once your client is set up, all the auth process is taken care of by itself, it is very straight forward. Example:payload = {...}endpoint_url = “https://api.cognitedata.com/api/v1/projects/my-project/context/diagram/detect/”response = client.post(endpoint_url, payload) When a payload is needed, you can refer to the documentation to get insights on its stru

Generate client secret for OID

Please generate a client secret once every 180 days.

Choose the application:

Badge winners

Andrew Montgomeryhas earned the badge Working with Cognite InField
Sahad Shaikhhas earned the badge Cognite Data Fusion Fundamentals
Flavio Costahas earned the badge Working with Cognite InField
Sai Meher Tejahas earned the badge Cognite Data Engineer Basics - Integrate
Flavio Costahas earned the badge Cognite Domain Expert Basics

Show all badges

Powered by Gainsight

Terms & Conditions Cookie Settings

Sign up

Already have an account? Login

Log in with your business account

Log in to the community

Log in with your business account

Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.

Enter your e-mail address

Back to overview

Scanning file for viruses.

Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.

OK

This file cannot be downloaded

Sorry, our virus scanner detected that this file isn't safe to download.

OK

RESOURCES Learn Documentation Support Status Page
GETTING STARTED Sign up to Cognite Hub Get Started with Cognite Hub About Us Contact Us
FOLLOW US

© 2022

Powered by inSided